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Overview 


This report summarizes the long-term findings of a rigorous random assignment evaluation of the 
WorkAdvance model, a sectoral training and advancement initiative. Launched in 2011, WorkAd- 
vance goes beyond the previous generation of employment programs by introducing demand-driven 
skills training and a focus on jobs that have identifiable career pathways. The model is heavily influ- 
enced by the positive findings from the Sectoral Employment Impact Study completed in 2010, as 
well as prior research on job retention and career advancement strategies. 


The WorkAdvance model was implemented between June 2011 and June 2013 by four providers — 
Per Scholas, St. Nicks Alliance, Madison Strategies Group, and Towards Employment — and a total 
of 2,564 individuals enrolled in the study. Several previous reports described the implementation, par- 
ticipation, cost, and interim impact findings of WorkAdvance and showed encouraging evidence for 
the WorkAdvance model. The impact findings presented in those reports covered the first three years 
of follow-up. While those findings showed earnings gains for some programs in some years, whether 
WorkAdvance could consistently increase earnings in the long term was still an open question. 


This report presents the long-term economic impacts of WorkAdvance and covers a two-year period 
occurring between four and eight years after individuals entered the study. The economic outcomes 
are based on National Directory of New Hires data and include 2017 and 2018. The report also builds 
on a previous cost analysis and presents findings from a full benefit-cost analysis to examine whether 
the effects of WorkAdvance resulted in gains or losses from the perspective of WorkAdvance partic- 
ipants, the government, and society. 


Key Findings 


e The WorkAdvance program at Per Scholas increased average earnings in 2017 and 2018; there 
were no statistically significant effects on average earnings at the other three sites. There is evi- 
dence that some of the WorkAdvance programs increased the likelihood of individuals having 
earnings of at least $30,000 in some time periods. None of the WorkAdvance sites increased em- 
ployment by a statistically significant amount in either long-term follow-up year. 


e In the pooled sample from all four providers, WorkAdvance had no effect on employment but 
increased average earnings and the likelihood of individuals having high earnings. 


e The overall pattern of economic impact findings suggests that the earnings-based impacts are 
driven by WorkAdvance group members having higher wages than control group members, rather 
than by being employed at a higher rate. This suggests WorkAdvance group members are advanc- 
ing over time, as intended by the WorkAdvance model. 


e The findings from the benefit-cost analysis are positive from the perspectives of the participants, 
the government, and society at all four sites. 


Overall, the WorkAdvance results support the case for focusing on how sector programs can be im- 
proved. The long-term economic impacts show that sector programs can increase earnings in the 
longer term and can lead to advancement gains over time for low-income individuals, but not all sector 
programs will lead to increases in employment and earnings. Focusing future efforts on how to make 
the sectoral approach — in particular, the advancement-focused services — more consistently suc- 
cessful can help workforce providers strengthen sector-based programs. This is the final planned re- 
port for the WorkAdvance evaluation. 
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Preface 


Over the past decade — in the same period that the WorkAdvance study has been conducted — 
the workforce development field has increasingly adopted the sectoral approach to meet the needs 
of both low-income workers and employers. Sector strategies train individuals for quality jobs in 
specific industries and occupational clusters where there is strong local demand and the oppor- 
tunity for career advancement. Although variations of sector strategies have been used since the 
1980s, interest in the approach grew after the release of the results from the Sectoral Employment 
Impact Study (SEIS) in 2010. Findings from that study, which used a rigorous random assignment 
design, showed positive earnings gains over a two-year follow-up period for individuals in three 
mature sector programs. 


After the release of that study’s findings, more workforce service providers started adopt- 
ing the sector approach. The WorkAdvance model and evaluation were developed starting in 
2011. WorkAdvance combined the most promising aspects of the programs involved in the SEIS 
evaluation — including strong employer relationships, a stringent screening process, and the pro- 
vision of individual, tailored services — with the best of what was known about advancement 
programs. The WorkAdvance evaluation sought to understand whether the overall approach 
could be a path to upward mobility for low-income individuals. Sector strategies also became a 
key component of the federal Workforce Innovation and Opportunity Act passed in 2014. 


To date, several studies, including WorkAdvance, have shown that sector strategies can 
be effective at helping people complete sector-based vocational training, obtain credentials and 
certifications, and find initial jobs within the target sectors. Less evidence is available on what 
happens to people after they start working in a given sector and whether they are able to move 
into higher-paying jobs, either by gaining new skills on the job or by obtaining additional training 
and certifications. One exception is an evaluation of Project QUEST, which has released eco- 
nomic impact findings through follow-up Year 9 and has shown that earnings gains can be sus- 
tained through a longer follow-up period. The findings from the WorkAdvance evaluation pre- 
sented in this report also add to that body of evidence and show that sector strategies can increase 
earnings in the longer term and lead to advancement gains over time for low-income individuals. 


As sector strategies continue to be adopted by workforce providers, the challenge will be 
to ensure that the approach can be effectively and consistently implemented across a range of 
providers; in particular, making sure that advancement-focused services are targeted and robust 
enough to help participants move up career pathways. Developing actionable evidence about how 
these programs can consistently lead to economic gains in different contexts, particularly in the 
long term, will be crucial to their success as a key strategy for upward mobility in the United 
States. 


Virginia Knox 
President, MDRC 
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Executive Summary 


Many individuals with low incomes struggle to obtain and maintain jobs that pay enough to meet 
their needs and put them on a path to upward mobility. At the same time, employers often report 
difficulty finding workers with the required skills. WorkAdvance, a workforce development 
model, seeks to overcome these challenges through a “dual customer” approach that meets the 
needs of both job seekers and employers. 


The WorkAdvance model was strongly influenced by prior research in two areas. First, 
it drew heavily on previous findings about sectoral strategies — strategies that train individuals 
for quality jobs in specific industries and occupational clusters where there is strong local demand 
and the opportunity for career advancement. The findings from one study in particular, the Sec- 
toral Employment Impact Study (SEIS), influenced the design of the WorkAdvance model.! It 
showed positive earnings gains over a two-year follow-up period for individuals in three mature 
sector programs. Second, WorkAdvance drew from earlier research on job retention and career 
advancement strategies. Results in this area have been mixed, but WorkAdvance is based on the 
hypothesis that concrete postemployment support — such as coaching tied to specific career 
paths and proactive reemployment services when a participant loses a job — may help individuals 
not only maintain their sector-based employment but also advance within the sector and continue 
to increase their earnings over time.? WorkAdvance sought to build on the SEIS findings and 
learn whether sector programs with an explicit focus on career advancement could be a path to 
upward mobility for low-income individuals. 


WorkAdvance Model and Evaluation 


The essential theory behind WorkAdvance is that offering low-income individuals education and 
employment-related skills and experience in high-demand sectors will help them advance in the 
labor market. This theory informs the five key components of the WorkAdvance model: 


1. Intensive screening of program applicants before enrollment for motivation 
and readiness, to ensure program providers select participants who can take 
advantage of the training and qualify for jobs in the target sector 


2. Sector-appropriate preemployment and career readiness services, including 
an orientation to the sector, career advancement coaching, and limited support 
services 


'Sheila Maguire, Joshua Freely, Carol Clymer, Maureen Conway, and Deena Schwartz., Tuning in to Local 
Labor Markets: Findings from the Sectoral Employment Impact Study (Philadelphia: Public/Private Ventures, 
2010). 

*Gayle Hamilton and Sue Scrivener, Increasing Employment Stability and Earnings for Low-Wage 
Workers: Lessons from the Employment Retention and Advancement (ERA) Project (New York: MDRC, 2012). 


3. Sector-specific occupational skills training aligned with employer needs, 
leading to certifications that are in demand in the regional labor market 


4. Sector-specific job development and placement services based on strong re- 
lationships with employers and intended to facilitate entry into positions that 
participants have been trained for and that offer genuine opportunities for con- 
tinued skills development and career advancement 


5. Postemployment retention and advancement services, including ongoing 
contact, coaching, skills training, and rapid reemployment help if needed 


The WorkAdvance model was implemented by four providers — Per Scholas, St. Nicks 
Alliance, Madison Strategies Group, and Towards Employment — and the programs were eval- 
uated using a randomized controlled trial design. (Table ES.1 provides an overview of the Work- 
Advance providers.) A total of 2,564 individuals enrolled in the study between June 2011 and 
June 2013 and were assigned at random to either the program (WorkAdvance) group or the con- 
trol group. Individuals in both research groups were tracked over time and their outcomes were 
compared to estimate the “impacts” of the programs. 


Table ES.1 


WorkAdvance Provider Characteristics 


Madison Strategies Towards 
Per Scholas St. Nicks Alliance Group Employment 
Location Bronx, NY Brooklyn, NY Tulsa, OK Northeast Ohio 
Information Environmental Transportation, Health care, 
Target sector(s) ae ; : 
technology remediation manufacturing manufacturing 
Sample size 690 479 697 698 


Several previous reports described the implementation, participation, cost, and interim 
economic impact findings of WorkAdvance and showed encouraging evidence for the WorkAd- 
vance model.* The impact findings presented in those reports covered the first three years of 


3In randomized controlled trial evaluations, these “impacts” can be attributed to the program, since the pro- 
gram and control groups are statistically alike at study entry and the only difference between them is that one 
group received program services and the other did not. 

‘Betsy Tessler, Michael Bangser, Alexandra Pennington, Kelsey Schaberg, and Hannah Dalporto, Meeting 
the Needs of Workers and Employers: Implementation of a Sector-Focused Career Advancement Model for Low- 
Skilled Adults (New York: MDRC, 2014); Richard Hendra, David H. Greenberg, Gayle Hamilton, Ari 
Oppenheim, Alexandra Pennington, Kelsey Schaberg, and Betsy L. Tessler, Encouraging Evidence on a Sector- 
Focused Advancement Strategy: Two-Year Impacts from the WorkAdvance Demonstration (New York: MDRC, 
2016); and Kelsey Schaberg, Can Sector Strategies Promote Longer-Term Effects? Three-Year Imacts from the 
WorkAdvance Demonstration (New York: MDRC, 2017). 


follow-up. While those findings showed earnings gains for some programs in some years, 
whether WorkAdvance could consistently increase earnings in the long term was still an open 
question. 


This report presents the long-term economic impacts of WorkAdvance and adds to the 
small body of rigorous evidence currently available on whether sector programs can increase em- 
ployment and earnings for low-income individuals beyond the third year after they enter such 
programs. The impacts are based on data collected from the National Directory of New Hires 
(NDNH) and cover 2017 and 2018, adding an additional two years of follow-up data for all sam- 
ple members.* Depending on when individuals entered the study, this two-year period occurred 
between four and eight years after they were randomly assigned. The analysis of the long-term 
data was done separately for each site.° 


The report also presents findings from a full benefit-cost analysis that examines whether 
the effects of WorkAdvance resulted in gains or losses from the perspectives of WorkAdvance 
participants, the government, and society. 


Key Findings 
Analyses in this report yielded the following key findings: 


e The WorkAdvance program at Per Scholas increased average earnings 
in both 2017 and 2018. At the other three sites, there were positive but 
not statistically significant differences in average earnings. However, the 
WorkAdvance programs at two of the other sites, in addition to Per 
Scholas, did increase the percentage of the sample with relatively high 
earnings. 


Per Scholas’s WorkAdvance program produced statistically significant impacts on aver- 
age earnings in both follow-up years. For example, in 2018, WorkAdvance group members 
earned an average of $38,404, an increase of $6,281 (or almost 20 percent) over the control group 
average (Table ES.2).’ Earnings gains of this size are rarely seen in random assignment studies 


5The findings presented in previous WorkAdvance reports used state unemployment insurance wage data 
to measure employment and earnings for all sample members. However, additional unemployment insurance 
wage data were not available for Madison Strategies Group sample members, so this report primarily focuses on 
data available through NDNH. 

°The effects of WorkAdvance were expected to strengthen as the programs gained more experience, and 
thus the economic impacts at each site were also analyzed by cohort — one of the study’s two prespecified, 
confirmatory subgroups. Sample members who came into the study during the first half of the intake period — 
between June 2011 and September 2012 — are in the “early cohort,” while the “late cohort” includes all remain- 
ing sample members, those who enrolled between October 2012 and June 2013. Findings from the cohort anal- 
ysis are presented in the main report. 

’The exhibits in the Executive Summary only show outcomes for 2018. This year was chosen because it 
was the latest year for which follow-up data were available. Outcomes for 2017 are discussed in the text and 
shown in the exhibits in the main report. 


Table ES.2 
Impacts on Employment and Earnings in 2018, by Site 


WorkAdvance Control Difference 


Group Group (Impact) P-Value 


Per Scholas 
Ever employed (%) 83.2 84.6 -1.4 0.624 
Total earnings ($) 38,404 32,122 6,281 *** 0.004 
Earned $30,000 or more (%) 57.0 50.7 6.3 * 0.099 


St. Nicks Alliance 


Ever employed (%) 77.7 79.7 = -2.0 0.601 
Total earnings ($) 26,670 23,822 2,849 0.208 
Earned $30,000 or more (%) 41.4 32.0 9.3 ** 0.033 


Madison Strategies Group 


Ever employed (%) 74.1 78.6 -4.4 0.168 
Total earnings ($) 21,248 20,461 787 0.603 
Earned $30,000 or more (%) 33.6 27.8 5.8 * 0.092 


Towards Employment 


Ever employed (%) 79.0 78.6 0.4 0.894 
Total earnings ($) 19,742 18,338 1,404 0.275 
Earned $30,000 or more (%) 28.9 23.8 5.1 0.117 


SOURCE: MDRC calculations from National Directory of New Hires data. 


NOTE: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 


of workforce programs and highlight the continued effectiveness of Per Scholas’s WorkAdvance 
program. 


At St. Nicks Alliance, Madison Strategies Group, and Towards Employment, WorkAd- 
vance group members earned more than control group members in both years on average, but the 
differences are not statistically significant (Table ES.2). 


Table ES.2 also shows that Per Scholas (by 6 percentage points), St. Nicks Alliance (by 
9 percentage points), and Madison Strategies Group (by 6 percentage points) increased the like- 
lihood of individuals having earnings of at least $30,000 in 2018. Madison Strategies Group also 
produced a statistically significant impact on this measure in 2017 (not shown).* Towards 


8Previous reports looked at impacts on the likelihood of individuals having earnings of at least $20,000 per 
year. That threshold was chosen based on the distribution of earnings for the pooled sample. Because the earnings 


Employment did not have a statistically significant effect on the likelihood of individuals having 
high earnings in either year, although more WorkAdvance group members than control group 
members had earnings of $30,000 or more in both years. 


e None of the WorkAdvance sites increased overall employment by a sta- 
tistically significant amount above the fairly high employment levels of 
the control groups in either 2017 or 2018. 


Across the sites, more than 78 percent of control group members worked in 2017 and 
2018, setting a high bar for the WorkAdvance programs. '° Table ES.2 shows that WorkAdvance 
group members and control group members worked at similar rates in 2018 at all four sites (a 
similar pattern is seen in 2017). The high employment rates among both research groups may 
reflect the low national unemployment rates during the follow-up years. 


e Pooling the samples from the four providers, WorkAdvance increased av- 
erage earnings and the likelihood of individuals having high earnings in 
2017 and 2018. WorkAdvance did not have a statistically significant effect 
on employment in either year for the pooled sample. 


The main WorkAdvance impact analysis was done at the site level. However, given the 
substantial variation in providers’ organizational emphases and prior experience operating sector 
strategies, it is also useful to understand how a model like WorkAdvance might perform, on av- 
erage, across a range of providers and contexts. Combining the sample from the four sites, Work- 
Advance had no effect on employment in either 2017 or 2018 (employment rates were high for 
both research groups in both years). However, WorkAdvance did increase earnings by statisti- 
cally significant amounts in both years for the pooled sample. In 2018, WorkAdvance increased 
earnings by $2,716 over the control group average (Table ES.3). Additionally, 40 percent of 
WorkAdvance group members had earnings of at least $30,000 that year, a statistically significant 
increase of 6 percentage points over the control group average. 


While the earnings impacts for the pooled sample show the effect for a range of possible 
WorkAdvance providers, they do mask the variation in impacts across the sites. In other words, 
some of the pooled sample earnings impacts are being driven by the large earnings impacts at Per 
Scholas. However, an exploratory analysis of the impacts for the pooled sample from only St. 
Nicks Alliance, Madison Strategies Group, and Towards Employment showed that the 


outcomes presented in this report are longer term than those presented in previous reports, and individuals tend 
to have higher earnings over time, this threshold was increased to $30,000. 

°Among the late cohort at Towards Employment, there was a statistically significant effect on the likelihood 
of individuals having earnings of $30,000 or more in 2017. 

‘0The employment outcomes reflect work in any sector, not just the ones the WorkAdvance sites targeted. 
WorkAdvance was designed to increase employment in the targeted sectors and not necessarily overall employ- 
ment. 


Table ES.3 


Impacts on Employment and Earnings in 2018 
for the Pooled Sample 


WorkAdvance Control Difference 
Outcome group group (Impact) P-Value 


Ever worked (%) 78.8 80.2 -1.5 0.355 
Total earnings ($) 26,419 23,703 92,716 *** 0.003 
Earned $30,000 or more (%) 40.0 33.7 6.4 *** 0.001 
Sample size 1,293 1,271 


SOURCE: MDRC calculations from National Directory of New Hires data. 


NOTE: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 


statistically significant earnings impacts remained in some long-term follow-up periods (not 
shown).!! This suggests that the earnings impacts of WorkAdvance were not limited to Per Scho- 
las. 


e The overall pattern of economic impacts suggests that WorkAdvance in- 
creased advancement: The earnings-based impacts are driven by Work- 
Advance group members having higher wages than control group mem- 
bers, rather than by being employed at a higher rate. This indicates that 
some WorkAdvance group members were advancing over time, as in- 
tended by the WorkAdvance model. 


Because WorkAdvance increased average earnings (at Per Scholas and for the pooled 
sample) without increasing overall employment, it can be inferred that WorkAdvance led to wage 
gains.” This is a sign that some WorkAdvance group members were advancing over time, as the 
model intended. Further evidence of advancement is seen in the statistically significant impacts 
on the likelihood of individuals having earnings of at least $30,000 at three of the four sites. 


'!This analysis indicated that among the pooled sample from St. Nicks Alliance, Madison Strategies Group, 
and Towards Employment, WorkAdvance increased earnings by statistically significant amounts in Quarter 3, 
2017 through Quarter 1, 2018; in Quarter 4, 2018; and in 2017 overall. 

The increases in earnings could also be driven by increases in hours worked. The NDNH data do not 
include information on hours worked, so it is not possible to test how much of the earnings impacts, if any, are 
attributable to hours worked. However, an analysis based on the Year 2 survey data showed that around half or 
more of WorkAdvance’s impact on eamings at each site was attributable to hourly wages (with the rest attribut- 
able to hours worked). 


e The findings from the benefit-cost analysis are positive from the perspec- 
tives of WorkAdvance participants, the government, and society at all 
four sites. 


As a result of increases in earnings and fringe benefits, WorkAdvance group members 
made substantial financial gains of between $5,500 and $15,500 during the combined observation 
and projection period (which ranges from 5 years to 10 years across the sites),'3 even though they 
paid higher taxes and relinquished appreciable amounts of government transfer benefits (Table 
ES.4). Although the government incurred considerable costs in operating WorkAdvance, these 
costs were at least offset at all four sites (and substantially so at Per Scholas) by participants pay- 
ing more in taxes and receiving less in government transfer benefits. Because participants were 
better off at all four sites and the government’s budget also improved, the financial gains for 
society at all four sites were substantial. Various sensitivity tests, including Monte Carlo simula- 
tions, produced similar findings. '* 


Conclusion 


WorkAdvance was an attempt to initially increase employment in promising sectors and eventu- 
ally increase earnings and help low-income individuals advance along a career pathway. Previous 
findings from the evaluation, which covered the first three years of follow-up, showed initial in- 
creases in employment in the targeted sector at all sites and increases in earnings at some sites. 
Long-term effectiveness, however, is a key measure of WorkAdvance and similar programs, 
given that it takes time for individuals to complete training, find initial sector jobs, and eventually 
advance into higher-level jobs within that sector. This report provides more evidence on the long- 
term effectiveness of the WorkAdvance programs. 


The findings show that the previous increases in employment seen at some of the sites 
faded in the long-term findings. None of the sites’ WorkAdvance programs increased overall 
employment by a statistically significant amount in either 2017 or 2018. However, the long-term 
impact findings show evidence of earnings increases at some sites. Per Scholas’s WorkAdvance 
program produced large impacts on average earnings in both 2017 and 2018. And three of the 
four WorkAdvance programs led to statistically significant increases in the likelihood of individ- 
uals having high earnings. Because there are increases in earnings and in high earnings without 


The observation period for Per Scholas and St. Nicks Alliance is 63 months and for Madison Strategies 
Group and Towards Employment, 62 months. The combined observation and projection period for Per Scholas 
is 7 years, and 10 years for St. Nicks Alliance. For Madison Strategies Group and Towards Employment, the 
benefits and costs were not projected beyond the 62-month observation period because it appears the positive 
earnings impacts disappeared after the observation period. 

‘4Results from the Monte Carlo analysis suggest that it is difficult to know for certain whether there were 
net losses or net gains from the government’s perspective at St. Nicks Alliance, Madison Strategies Group, and 
Towards Employment. Whatever the direction, they were probably small, suggesting that program operating 
costs were largely offset. See Chapter 3 of the main report for more information. 


Table ES.4 


Benefits and Costs by Accounting Perspective Over 
the Observation Period and for Two Sites Over the 
Observation and Projection Periods (in 2018 Dollars), by Site 


Government 

Net financial gains and losses ($) Participants Budget Society 
Per Scholas 

Over the observation period 13,997 11,370 27,535 

Over the observation and projection periods 15,456 13,387 31,387 
St. Nicks Alliance 

Over the observation period 1,623 -4,130 -3,293 

Over the observation and projection periods 9,387 3,660 13,742 
Madison Strategies Group 

Over the observation period 11,192 1,615 13,114 

Over the observation and projection periods 11,192 1,615 13,114 
Towards Employment 

Over the observation period 5,505 265 5,820 

Over the observation and projection periods 5,505 265 5,820 


SOURCES: The sources and derivation of net program costs are described in Hendra et al. 
(2016). The sources and derivation of the remaining benefit and cost components are described 
in Appendix B. 


NOTES: The observation period for Per Scholas and St. Nicks Alliance is 63 months and for 
Madison Strategies Group and Towards Employment is 62 months. The combined observation 
and projection period for Per Scholas is 7 years and for St. Nicks Alliance is 10 years. For 
Madison Strategies Group and Towards Employment, the benefits and costs were not projected 
beyond the observation period because it appears the positive impacts on earnings 
disappeared after the observation period. 

All gains and losses include the monetized effects of WorkAdvance on nonmarket time and 
deadweight loss, are inflation-adjusted to 2018 dollars, and are discounted to 2018 present 
values. 


commensurate increases in employment, there is evidence that WorkAdvance did lead 
to wage gains. This is a sign that WorkAdvance group members were advancing over 
time, as the model intended. 

The findings from the benefit-cost analysis show large financial gains from the perspec- 
tives of WorkAdvance participants and society at large for all four sites. The size of these gains 
is exceptional when compared with benefit-cost findings from other evaluations of employment 
and training programs. 


Overall, the WorkAdvance results reinforce other rigorous research that shows that sec- 
toral programs can be quite effective; the results also support the case for investigating why 
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certain types of programs are successful and how they can be improved. The long-term economic 
impacts of WorkAdvance show that sector programs can increase earnings in the longer term and 
can lead to advancement gains over time for low-income individuals. At the same time, sector 
programs are difficult to design and implement well, and not all programs will lead to statistically 
significant increases in employment and earnings. Focusing future efforts on how to make the 
sectoral approach more consistently successful will help workforce providers effectively imple- 
ment or strengthen such programs. 


This is the final planned report for the WorkAdvance evaluation. Longer-term findings 
from several other evaluations of sector programs will be released in the next few years, which 
will provide more evidence on the effectiveness of sector strategies. 


Chapter 1 


Introduction 


Many individuals with low incomes struggle to obtain and maintain jobs that pay enough to meet 
their needs and put them on a path to upward mobility. At the same time, some employers report 
difficulty finding workers with the required skills. WorkAdvance, a workforce development 
model, seeks to overcome these challenges through a “dual customer” approach that meets the 
needs of both job seekers and employers. The WorkAdvance model was strongly influenced by 
prior research in two areas: sector strategies, and job retention and career advancement services. 


Sector strategies train individuals for quality jobs in specific industries and occupational 
clusters where there is strong local demand and the opportunity for career advancement. The 
findings from one study — the Sectoral Employment Impact Study (SEIS), completed by Pub- 
lic/Private Ventures in 2010! — motivated some of the core aspects of the WorkAdvance model. 
That study was the first rigorous test of sector strategies, and the findings showed positive earn- 
ings gains over a two-year follow-up period for individuals in three mature sector programs.” The 
encouraging findings from that study created significant interest in sector programs. There has 
been a proliferation of new programs in recent years, and sector strategies were a key component 
of the federal Workforce Innovation and Opportunity Act passed in 2014. 


WorkAdvance also draws on lessons from efforts to improve job retention and career 
advancement of low-skilled workers after initial job placement. Retention and advancement pro- 
grams have had mixed results, but much has been learned about what is likely to be effective and, 
equally important, ineffective. Particularly relevant for WorkAdvance is the hypothesis that con- 
crete postemployment support — such as coaching tied to specific career paths and proactive 
reemployment services when a participant loses a job — could help individuals not only maintain 
their employment, but also continue to increase their earnings over time.* 


WorkAdvance sought to build on the SEIS findings and learn whether sector programs 
with an explicit focus on career advancement could be a path to upward mobility for low-income 
individuals. The WorkAdvance model was implemented by four providers, and the programs 
were evaluated using a randomized controlled trial design. A total of 2,564 individuals enrolled 
in the study between June 2011 and June 2013 and were assigned at random to either the program 


'Maguire et al. (2010). 

The three programs in the SEIS had all been operating for at least three years before the evaluation. This 
was not the case for the WorkAdvance providers. 

3Hamilton and Scrivener (2012). 


(WorkAdvance) group or the control group. Individuals in both research groups were tracked 
over time, and their outcomes were compared to estimate the “impacts” of the programs.‘ 


Several previous reports described the implementation, participation, cost, and interim 
impact findings of WorkAdvance.° The impact findings presented in those reports covered the 
first three years of follow-up. While those findings showed earnings gains for some programs in 
some years, whether WorkAdvance could consistently increase earnings in the long term was still 
an open question. 


Several other evaluations of sector programs, including a few ongoing evaluations, have 
released findings since the WorkAdvance evaluation started in 2011. Some of them have also 
found encouraging evidence for sector strategies. However, most of the currently available evi- 
dence is on the short- to medium-term effectiveness of these strategies (through no more than 
three years of follow-up). One exception is an evaluation of Project QUEST, which has released 
economic impact findings through follow-up Year 9. Findings from that evaluation show earnings 
gains for individuals in a health care-focused program in Years 4, 5, 6, and 9.° There are other 
ongoing evaluations that have released short-term findings to date. The Health Profession Oppor- 
tunity Grants (HPOG) evaluation’ is examining programs using a demand-driven training ap- 
proach focused on the health care sector. The Pathways for Advancing Careers and Education 
(PACE) demonstration® is evaluating several programs using a sector-focused career pathways 
approach. Both will be releasing longer-term findings in the next few years. 


This report presents the long-term economic impacts of WorkAdvance covering a two- 
year period occurring between four and eight years after individuals entered the study. It adds to 
the small body of rigorous evidence currently available on whether the sector approach can in- 
crease employment and earnings for low-income individuals beyond the third year after they enter 
such programs. The report also builds on a previous cost analysis and presents findings from a 
full benefit-cost analysis to examine whether the effects of WorkAdvance resulted in gains or 
losses from the perspective of participants, the government, and society. 


WorkAdvance Model 


The essential theory behind WorkAdvance is that education and employment-related skills and 
experience in high-demand sectors will eventually lead to advancement in the labor market. This 
theory informs the five key components of the WorkAdvance model (depicted in Figure 1.1): 


‘Tn randomized controlled trial evaluations, these “impacts” can be attributed to the program, since the pro- 
gram and control groups are statistically alike at study entry and the only difference between them is that one 
group received program services and the other did not. 


5Tessler et al. (2014); Hendra et al. (2016); and Schaberg (2017). 
®Roder and Elliott (2019). 

Peck et al. (2018). 

Gardiner and Juras (2019). 


Figure 1.1 


WorkAdvance Model Components 


Occupational Job 

skills training 

leading to a 
credential 


Retention and 
advancement 
services 


Career 
readiness 
services 


Intensive 


screening development 


and placement 


1. Intensive screening of program applicants before enrollment for motivation and 
readiness to ensure that program providers select individuals who can take advantage 
of the training and be qualified for jobs in the target sector 


2. Sector-appropriate preemployment and career readiness services, including an 
orientation to the sector, career advancement coaching, and limited support services 


3. Sector-specific occupational skills training aligned with employer needs and lead- 
ing to certifications that are in demand in the regional labor market 


4. Sector-specific job development and placement services based on strong relation- 
ships with employers and intended to facilitate entry into positions that participants 
have been trained for and that are thought to offer genuine opportunities for continued 
skills development and career advancement 


5. Postemployment retention and advancement services including ongoing contact, 
coaching, skills training, and rapid reemployment help if needed 


Findings from Previous Reports 


The WorkAdvance model was implemented by four providers specializing in specific sectors in 
which they have sought to develop relationships with employers and in-depth industry 
knowledge: Per Scholas (in New York City) targeted the information technology (IT) sector; St. 
Nicks Alliance (also in New York City) focused on environmental remediation and related occu- 
pations; Madison Strategies Group (in Tulsa, Oklahoma) focused on transportation and, later, 
manufacturing; and Towards Employment (in northeast Ohio) targeted health care and manufac- 
turing. 


Previous findings from the evaluation’s implementation, participation, cost, and interim 
economic impact analyses all showed encouraging evidence for the WorkAdvance model. These 
analyses were done at the site level because of substantial variation in prior experience operating 


sector strategies and organizational emphases across the four providers. Table 1.1 provides a sum- 
mary of the key features and sample composition of each site.? WorkAdvance targeted unem- 
ployed and low-wage workers with a family income below 200 percent of the federal poverty 
level, but there was variation in the characteristics of individuals who ultimately enrolled in the 
study both across and within sites. The providers chose which sectors and occupations to target 
based on their own experience, local labor market demand, and the potential for advancement 
within the sector. Some of the providers shifted their specific training, curriculum, or credential 
offerings partway through the study period in response to employer needs and changes in their 
local labor market. 


Implementation Analysis 


One of the main findings from the implementation analysis was that it took time for the 
providers — especially those that had not operated a sector-focused program previously — to 
fully implement all the WorkAdvance model components. Because of this, individuals who en- 
tered the study later probably received a stronger set of services than individuals who came in 
earlier. It was hypothesized that because of this difference in the maturity of the programs and 
their services over time, the impacts for individuals who entered the study later would be larger 
than the impacts for individuals who entered the study earlier. 


Another key piece of the implementation story was that two of the providers — Towards 
Employment and Madison Strategies Group — initially implemented a “‘placement first” track, 
in which some participants skipped occupational skills training and sought immediate employ- 
ment,!° while other participants followed the main “training first” track as outlined in Figure 1.1. 
About halfway through the study enrollment period, the placement-first track was phased out at 
both sites after preliminary evidence showed that individuals in that track were entering low-wage 
jobs and were not gaining the skills needed to advance. This change in the type of services par- 
ticipants received is another reason why the impacts were hypothesized to be stronger for late 
study enrollees than for early study enrollees at these two sites. 


Cost Analysis 


WorkAdvance services cost between $6,400 and $8,300 per participant (in 2018 dollars) 
across the four providers, and for three of the providers (excluding St. Nicks Alliance), the range 
was quite narrow — $6,400 to $6,800.'' Roughly half of the providers’ operating expenditures 
were devoted to providing preemployment activities and occupational skills training. These gross 
costs are simply the outlays required to operate the programs. 


°As MDRC uses it, “site” is short for “experimental site,” a term that encompasses the program, the Work- 
Advance group, the control group, and the local environment. 

‘The placement-first track was intended to be a less expensive but still effective route to advancement. The 
idea was that individuals would gain experience and sector-specific skills (through on-the-job training, for ex- 
ample) without going to formal training first. Another rationale for the track was that it would help the providers 
build relationships with employers sooner, because they were able to offer and deliver a more immediate service. 


‘Tn Hendra et al. (2016), the costs of WorkAdvance were reported in 2013 and 2014 dollars. 


Table 1.1 
WorkAdvance Providers and Sample Composition at Baseline 


Madison Towards 
Per Scholas St. Nicks Alliance Strategies Group Employment 
Provider characteristics 

Location Bronx, NY Brooklyn, NY Tulsa, OK Northeast Ohio 
Information Environmental Transportation, Health care, 

Target sector(s) ie : , 
technology remediation manufacturing manufacturing 
Training and Training and 
iss ae placement first until placement first until 
Approach Training first Training first fall 2012: then fall 2012: then 


mostly training first mostly training first 
Sample composition 


Average age 31 35 35 35 
Female (%) 13 15 16 59 
Some college 63 AA 58 57 
or more (%) 

Currently employed (%) 13 11 27 27 
Ever employed (%) 96 98 99 97 
Received food 

stamps/SNAP (%) ud ae 25 2 
Previously convicted of 10 20 40 25 


a crime (%) 


SOURCES: Information from documentation supplied by providers and MDRC calculations from the WorkAdvance 
baseline information form. 


Net cost figures are the extra costs society incurred by letting eligible individuals attend 
WorkAdvance rather than what they would have done otherwise, in the absence of the program. 
Because WorkAdvance encompasses activities that individuals targeted by the program might 
have engaged in on their own (as some of the control group members did), the net cost was about 
$3,750 per WorkAdvance group member for Per Scholas and in the range of $5,100 to $6,300 at 
the other three sites (these net costs are in 2018 dollars). This suggests that in the absence of 
WorkAdvance, very little would have been spent to provide training or other services to the indi- 
viduals who were interested in such a program. 


Participation Analysis 


WorkAdvance group members were eligible to receive all the services provided in the 
WorkAdvance programs: career readiness, occupational skills training, job search, and postem- 
ployment services. Control group members, on the other hand, were not eligible to recetve Work- 
Advance services, although they were free to seek out other services on their own in their com- 
munities. The study’s “treatment contrast” can be measured by comparing the rate of service 
receipt among WorkAdvance group members to the rate of service receipt among control group 


members. The findings showed that WorkAdvance at all four sites produced large increases in 
participation, relative to the control group, in all the model components. Notably, WorkAdvance 
increased individuals’ likelihood of completing occupational skills training in the targeted sector 
by 31 percentage points or more, compared with the control group rates at every site. It also in- 
creased the likelihood of individuals obtaining a credential in that sector by between 25 and 46 
percentage points across the sites. This level of increase in service receipt is not often seen in 
workforce programs, since program enrollees often have barriers that prevent them from fully 
engaging in and completing services, especially in occupational skills training that can last for 
several months. These large participation increases present a good test of whether the services 
offered by WorkAdvance are effective in increasing economic outcomes for low-income individ- 
uals beyond what would have happened without the program. 


Economic Impact Analysis 


Previous reports presented the economic impacts of WorkAdvance based on a survey 
administered roughly two years after individuals entered the study (called the “Year 2 Survey’’) 
and based on administrative data through three years of follow-up.” In general, the previous im- 
pact findings varied across the sites. 


All the sites increased employment in the targeted sector as measured by the Year 2 Sur- 
vey. However, this was not a sufficient condition for impacts on overall employment, earnings, 
and advancement. Per Scholas produced large impacts on employment and earnings, as well as 
on several secondary outcomes, including measures of advancement, income, and life satisfac- 
tion. The economic impacts at the site grew stronger throughout the three-year follow-up period. 
St. Nicks Alliance did not produce any impacts on employment or earnings throughout most of 
the three-year follow-up period. Madison Strategies Group produced impacts on earnings among 
the late cohort in Years 2 and 3. Among the full site sample, the site increased earnings in Year 
2, but the impact faded in Year 3. The site also increased wages above $15 per hour and produced 
impacts on several measures of employer-offered benefits (for example, an increase in the avail- 
ability of health insurance and paid vacations). Towards Employment increased earnings among 
both the late cohort and the full site sample in Year 2, but the impacts faded among both samples 
by Year 3. The site produced positive impacts on some measures of nonfinancial advancement 
and work schedules. 


Roadmap for the Report 


This report builds on the previous WorkAdvance findings in a few ways. First, it extends the 
follow-up period for the economic impact analysis by two years. The long-term employment and 
earnings impacts of WorkAdvance, which cover a two-year period occurring between four and 
eight years after individuals entered the study, are based on data collected from the National Di- 
rectory of New Hires. Findings are presented by WorkAdvance site, for the pooled sample 


The administrative data findings were based on state unemployment insurance wage data. 


(combining sample members from all four sites), and for the study’s two confirmatory subgroups: 
random assignment cohort and level of labor market attachment (Chapter 2). 


The report also presents findings from a benefit-cost analysis of WorkAdvance. For each 
site, benefits and costs are presented to see whether, when viewed in aggregate, the WorkAdvance 
programs resulted in gains or losses. Benefits and costs are presented from the perspectives of 
WorkAdvance participants, the government, and society (Chapter 3). 


Finally, the report summarizes the economic impact findings from the WorkAdvance 
evaluation — both in the long term and over the full follow-up period — and provides suggestions 
for what future research efforts in this area should focus on (Chapter 4). 


In brief, the report shows that the long-term effects of WorkAdvance vary across the four 
providers. Per Scholas’s WorkAdvance program increased average earnings by a large amount 
in the long term but had no effect on employment. WorkAdvance at St. Nicks Alliance did not 
have a statistically significant effect on employment or average earnings in the long term. There 
is some evidence the program increased the likelihood of individuals having high earnings 
($30,000 or more).'* Among both the late cohort and full sample at Madison Strategies Group, 
there were no statistically significant effects on employment or average earnings, but the program 
did increase the likelihood of individuals having high earnings among both samples. Similarly, 
WorkAdvance at Towards Employment had no statistically significant effects on employment or 
average earnings among the late cohort or full sample. There is some evidence that the program 
there increased the likelihood of having high earnings among the late cohort. 


The report also shows that WorkAdvance resulted in financial gains from the perspec- 
tives of WorkAdvance participants, the government, and society at all four sites. Due to increases 
in earnings and fringe benefits, the gains for WorkAdvance participants ranged from $5,000 to 
$15,000 over a 5- to 10-year period across the sites. These gains occurred despite participants 
paying more in taxes and receiving less in government transfer benefit payments. The costs to the 
government of operating WorkAdvance were at least offset at every site — and at some sites, 
were more than offset — by the increases in tax payments made by and reductions in transfer 
benefits paid to WorkAdvance participants. '* And because participants were better off at all four 
sites and the government at least broke even, the financial gains for society were also positive at 
all four sites. These positive benefit-cost findings are not often seen in evaluations of employment 
and training programs. 


Previous reports looked at impacts on the likelihood of individuals having earnings of at least $20,000 per 
year. That threshold was chosen based on the distribution of earnings for the pooled sample. Because the earnings 
outcomes presented in this report are longer term than the outcomes presented in previous reports, and individuals 
earnings tend to increase over time, the threshold was increased to $30,000 in this report. 

‘4s discussed in Chapter 3, results from the Monte Carlo analysis suggest that it is difficult to know for 
certain whether there were net losses or net gains from the government’s perspective at St. Nicks Alliance, Mad- 
ison Strategies Group, and Towards Employment, although whatever the direction, they were probably small. 


Chapter 2 


Long-term Economic Impact Findings 


Since the last report on WorkAdvance, additional administrative data on employment and earn- 
ings were collected from the National Directory of New Hires (NDNH). The findings presented 
in previous WorkAdvance reports used state unemployment insurance wage data to measure em- 
ployment and earnings for all sample members. However, additional unemployment insurance 
wage data were not available for Madison Strategies Group sample members, and thus this report 
primarily focuses on data available through NDNH. See Box 2.1 and Appendix A for more in- 
formation on the previous findings, as well as the longer-term findings based on state unemploy- 
ment insurance wage data for sites where these data were available. 


As with the previous reports, the analysis of the long-term data was done at the site level. 
Because the effects of WorkAdvance were expected to strengthen as the programs gained more 
experience, the economic impacts at each site were also analyzed by cohort — one of the study’s 
two prespecified, confirmatory subgroups. ! Sample members who came into the study during the 
first half of the intake period — between June 2011 and September 2012 — are in the “early 
cohort,” while the “late cohort” includes all remaining sample members, those who enrolled be- 
tween October 2012 and June 2013. Figure 2.1 shows the sample enrollment periods and sample 
sizes for the full sample, as well as for the early and late cohort samples, at each site. 


The NDNH data cover 2017 and 2018, adding an additional two years of follow-up data 
for all sample members.” These calendar years correspond to different relative years of follow-up 
data for different sample members, depending on when they entered the study (see Figure 2.2). 
For individuals who entered the study in the beginning of the sample enrollment period (in mid- 
2011), the NDNH data cover the first half of relative Year 6 to the first half of relative Year 8. 
For individuals who entered the study at the end of the enrollment period (in mid-2013), the 
NDNH data cover the first half of relative Year 4 to the first half of relative Year 6. In other 
words, the impact analysis in this report covers sample members’ labor market outcomes during 
a two-year period that occurs between four and eight years after random assignment, depending 
on when participants entered the study. 


‘Tn recent years, the program evaluation field has become more sensitive to the need to limit the number of 
subgroups analyzed in order to reduce the number of “false positives” that result when one makes too many 
statistical comparisons. In order to manage this risk, methodologists have recommended prespecification (mean- 
ing the subgroups are chosen before estimating any impacts) of a limited number of “confirmatory” subgroups 
that theory and experience suggest might moderate the impacts of a program (see Bloom and Michalopoulos, 
2010). The other prespecified, confirmatory subgroup analysis is based on sample members’ levels of attachment 
to the labor market at study entry. Findings from that analysis are presented in a later section. 

Only the eight most recent quarters of NDNH employment and earnings data are available at any given 
time. All previous quarters of data “roll off’ and are not available for data matching or analysis. 


3Relative years are based on each sample member’s quarter of random assignment. For example, “Year 1” 
refers to the first four quarters following each sample member’s quarter of random assignment. 


Box 2.1 
State Unemployment Insurance Wage Data 


Previous WorkAdvance reports presented employment and earnings impacts based on state un- 
employment insurance wage data through Year 3.* For this report, the evaluation team was able 
to collect additional state unemployment insurance wage data from New York (covering sample 
members from Per Scholas and St. Nicks Alliance) and Ohio (covering sample members from 
Towards Employment) through Year 5. However, the evaluation team was not able to collect 
additional state unemployment insurance wage data from Oklahoma, so data are only available 
through Year 3 for sample members from Madison Strategies Group. 


Impacts based on state unemployment insurance wage data are presented in Appendix Tables 
A.2 to A.5. In general, the findings tell a story that is similar to the NDNH-based findings: 


* Per Scholas produced impacts on employment through Year 3 and on earnings through 
Year 5. 


St. Nicks Alliance had no economic effects in Years | through 3; starting in Year 4, the 
site produced larger increases in earnings, although the differences were not statistically 
significant. 


Madison Strategies Group increased earnings among the late cohort in Years 2 and 3 (it is 
unclear whether this pattern continued in later years). Among the full sample, the site in- 
creased earnings in Year 2, but the effect faded in Year 3. 


Towards Employment increased earnings by statistically significant amounts in Years 2 
and 4 among the late cohort, although the effect faded again in Year 5. Among the full 
sample, the site increased earnings in Year 2, but the effect faded by Year 3. 


NOTES: 

*Findings based on state unemployment insurance wage data are presented relative to each sample 
member’s quarter of random assignment. For example, “Year 1” refers to the first four quarters following 
each sample member’s quarter of random assignment. 


Based on the model, it was hypothesized that the WorkAdvance programs would first 
increase employment, primarily in the targeted sector, and then increase earnings. The WorkAd- 
vance providers sought to place individuals in jobs with opportunities for advancement, rather 
than in just any job. So, while control group members would probably find employment on their 
own (or with the help of other organizations in the community), the idea was that WorkAdvance 
group members would be more likely than control group members to obtain good jobs in the 
targeted sector. These jobs were also expected to have higher initial wages and more opportunities 
for individuals to continue developing the skills needed to move into more advanced positions 
(leading to increases in earnings over time) than the jobs obtained by control group members. 
Further, the advancement-focused services provided by WorkAdvance, including those provided 
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Figure 2.1 


Study Enrollment Period and Sample Size, 
by Site and Analysis Sample 


June February October June 
2011 2012 2012 2013 
Total sample 
Per Scholas 690 
St. Nicks Alliance 479 
Madison Strategies Group 697 
698 


Towards Employment 


Early cohort Late cohort 
SOURCE: MDRC calculations from WorkAdvance baseline information form. 
NOTE: The colored bars reference time periods. Per Scholas began study enrollment in June 2011, St. Nicks 
Alliance began study enrollment in September 2011, and Madison Strategies Group and Towards Employment 


began study enrollment in October 2011. The numbers within the colored bars indicate the sample sizes for the 
early and late cohorts at each site. 


Figure 2.2 


Comparison of Calendar Years Covered by National Directory of New Hires Data 
to Relative Years, by Quarter of Random Assignment and Cohort 


2017 2018 
Study enrollment date Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 


June 


2011 


July to September 
October to December 


January to March 
April to June 
July to September 


2012 


October to December 


January to March 
April to June Year 4 


Late 
cohort 


2013 


SOURCE: MDRC calculations from WorkAdvance baseline information form. 
NOTE: The NDNH data cover 2017 and 2018. This time period corresponds to different relative years depending on 


when individuals entered the study. For example, for individuals who entered the study at the end of the enrollment 
period (from April to June 2013), the NDNH data cover the first half of relative Year 4 to the first half of relative Year 6. 
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postemployment, were designed to help individuals plan for and navigate the next steps along a 
career pathway. The length of follow-up data available in this report — between Year 4 and Year 
8 — allows for a test of whether the WorkAdvance programs led to these advancement gains, as 
measured by earnings. 


In summary, this report’s findings indicate that WorkAdvance’s long-term economic im- 
pacts still vary across providers. There is no evidence that WorkAdvance increased employment 
in the long term at any of them, but there is evidence of earnings increases at some sites. Per 
Scholas’s WorkAdvance program increased average earnings in both 2017 and 2018 and in- 
creased the likelihood of individuals having earnings of at least $30,000 in 2018. St. Nicks Alli- 
ance’s WorkAdvance program did not have a statistically significant effect on average earnings 
in either year. However, the program increased the likelihood of individuals having earnings of 
at least $30,000 in 2018. Among both the late cohorts and full samples at Madison Strategies 
Group and Towards Employment, WorkAdvance produced no effects on average earnings in 
either long-term follow-up year. Madison Strategies Group’s program, however, increased the 
likelihood of having high earnings in both years among the full sample and in 2017 among the 
late cohort. Among the late cohort at Towards Employment, WorkAdvance increased the likeli- 
hood of having high earnings in 2017. Because there are no impacts on employment across the 
sites, but there are some impacts on earnings, there is evidence that WorkAdvance group mem- 
bers were advancing into higher-wage jobs over time, a key goal of the WorkAdvance model. 


The next section provides a summary of economic conditions during the study period and 
how those conditions may have interacted with the study’s findings. The later sections provide an 
overview of the WorkAdvance providers and detail economic impacts for each one. Box 2.2 ex- 
plains how to read the impact tables in this report. 


Economic Conditions During the Study Period 


WorkAdvance study enrollment started in mid-2011 and went through mid-2013, with individu- 
als in the late cohort receiving services through mid-2015. This study period coincided with the 
slow recovery following the Great Recession of 2007 to 2009, when even relatively experienced 
and skilled workers struggled to find employment. The early part of the recovery was notable for 
its lack of job creation and earnings growth. This was compounded by the fact that the period up 
to 2007 was sometimes called the “jobless recovery” from the recession in the early 2000s. Thus, 
low-wage workers confronted an extended period of labor market stagnation.* Studies indicate 
that employers responded to this increased supply of unemployed workers by being especially 
selective about whom they hired, particularly in relation to recent work experience. Those who 
were out of the labor market for six months or longer were much less likely to receive calls for 
job interviews, even when they had extensive relevant experience.° 


4Kolesnikova and Liu (2011). 
5Ghayad (2013); Kroft, Lange, and Notowidigdo (2013). 
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Box 2.2 
How to Read the Tables in This Report 


Most tables in this report use a similar format, illustrated below. The table shows the em- 
ployment and earnings outcomes for the WorkAdvance and control groups at Per Scholas. 
For example, the table shows that 85 percent of the site’s WorkAdvance group members 
ever worked in 2017, compared with 83 percent of control group members. 


Because study participants were assigned randomly to either the WorkAdvance group or the 
control group, the effects of WorkAdvance can be estimated by the difference in outcomes 
between the two groups. The “Difference” column in the table shows the WorkAdvance 
group’s employment and earnings outcomes minus the control group’s employment and 
earnings outcomes; in other words, WorkAdvance’s impact on employment and earnings. 
For example, the impact on earnings in 2017 is calculated by subtracting $28,049 from 
$32,552, yielding $4,503. 


The “P-value” column gives an indication of how unlikely it is that the impact arose by 
chance. The lower the p-value, the less likely it is that the impact arose by chance. Impacts 
are considered statistically significant if they have a p-value below 0.100, meaning there is 
less than a 10 percent chance that the impact arose by chance (or in other words, meaning 
there is less than a 10 percent chance that the true impact is zero or even negative). Statisti- 
cally significant differences are marked with asterisks. The number of asterisks indicates 
whether the impact is statistically significant at the 1 percent, 5 percent, or 10 percent level 
(the lower the level, the more asterisks). For example, the p-value for the impact on earnings 
in 2017 is 0.027. This indicates that there is a 3 percent chance of observing an impact of at 
least $4,503 if Per Scholas’s WorkAdvance program really had no true effect on earnings 
that year. Two asterisks indicate that this impact is statistically significant at the 5 percent 
level (this is also seen by the p-value being below the 0.050 threshold). 


Employment and Earnings Impacts, Per Scholas 


WorkAdvance Control Difference 
Outcome rou rou Impact P-value 


Ever worked (%) 
2017 84.5 83.3 1.3 0.651 
2018 83.2 84.6 -1.4 0.624 


Total earnings ($) 
2017 32,552 28,049 4,503 0.027 
2018 38,404 32,122 6,281 0.004 


In recent years there have been increasing concerns about p-values and statistical signifi- 
cance being overinterpreted or used in isolation to interpret research findings. As noted in 
guidance provided by the American Statistical Association, “Statistical significance is not 
equivalent to scientific, human, or economic significance. Smaller p-values do not neces- 
sarily imply the presence of larger or more important effects, and larger p-values do not 
imply a lack of importance or even lack of effect.”’* 


*Wasserstein and Lazar (2016). 
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These labor market conditions may have had a greater impact on individuals who entered 
the WorkAdvance study earlier — sooner after the end of the Great Recession — than those who 
entered later. Research has shown that there are large, negative effects on wages for college grad- 
uates who enter the labor market during an economic downturn, and those effects can persist for 
decades.° This may explain at least some of the overall pattern in the cohort findings: Across the 
sites, the 2017 and 2018 earnings levels for the control group, and in most cases, for the Work- 
Advance group, were higher among the late cohort than among the early cohort. Individuals in 
the late cohort may have been able initially to obtain better jobs with higher wages than those in 
the early cohort, starting them off at a higher point and perhaps putting them on a better path 
toward upward mobility. 


The economic outcomes presented in this report cover 2017 and 2018. The economy was 
fairly strong, especially relative to previous years, during this two-year period. By early 2017, the 
unemployment rate had returned to prerecession levels and it dropped even lower in 2018 (to 3.7 
percent in September 2018, nationally).”? With lower unemployment rates, it may have been easier 
for both WorkAdvance group and control group members to find employment. 


Per Scholas 


Per Scholas, a nonprofit organization that provides information technology training and employ- 
ment services in New York City, came into the study with substantial experience operating a 
sector program. The organization had been operating most of the WorkAdvance model compo- 
nents, except for the advancement-focused and postemployment services, since 1998, and was 
able to adapt its curriculum and training offerings based on employer feedback. This experience 
gave Per Scholas a head start over some of the other providers who were newer to the model,® 
and its advantage is evident in the economic impact findings for the site. Box 2.3 provides an 
update on Per Scholas’s program since the original WorkAdvance evaluation ended. 


Per Scholas increased earnings by statistically significant amounts in both 2017 and 2018 
(Table 2.1). In 2018, WorkAdvance group members earned an average of $38,404, an increase 
of $6,281 (or almost 20 percent) over the control group average.’ Earnings gains of this size are 
rarely seen in random assignment studies of workforce programs and highlight the continued 
effectiveness of Per Scholas’s WorkAdvance program. 


®Kahn (2010). 
U.S. Bureau of Labor Statistics (2019a). 
8Per Scholas had also previously participated in another randomized controlled trial, the SEIS. 


°The employment and earnings levels of both the WorkAdvance group and the control group at Per Scholas 
are higher than those at the other three WorkAdvance sites. 
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Box 2.3 
Where Are the WorkAdvance Providers Now? 


Per Scholas 


Since the original WorkAdvance evaluation ended in late 2014, Per Scholas has expanded 
its program nationwide and now operates training programs in eleven cities: Atlanta, Balti- 
more, Boston, Cincinnati, Columbus, OH, Dallas, the DC region, Detroit, New York, New- 
ark, NJ, and Philadelphia. The organization has been focused on continuing to grow in terms 
of both the number of people it trains and the services it offers. 


Per Scholas has stayed on top of trends in the information technology (IT) industry and the 
needs of the employers it works with. The organization has expanded its IT training curric- 
ulum and now offers training in other specialties, including cybersecurity and software en- 
gineering. To reach more students, the organization has also developed bridge programs for 
applicants who do not meet the program’s required math and reading levels. These bridge 


programs provide support services, teach basic level education and technical skills, and 
serve as a pipeline into the main Per Scholas training. 


As part of the WorkAdvance evaluation, Per Scholas added advancement-focused and 
postemployment services and has continued to develop them since the evaluation ended. 
For example, the organization now has students engage with employers earlier in the pro- 
gram — around three to four weeks after they start the training — so they can develop their 
social skills and get a glimpse of what they can expect after the program ends. Per Scholas 
also continues to track and reach out to students at regular intervals for two years after they 
graduate from the program. 


Per Scholas is also innovating in the types of services it offers. The organization has started 
offering customized trainings that teach more advanced IT skills and are developed through 
a close partnership with and funded by employers. The idea is that this will serve as a talent 
pipeline for the employer. Per Scholas has also recently started exploring how they can use 
remote learning to reach more students and how they can develop an alumni leadership 
academy that further develops leadership skills in program graduates. 


At the same time, Per Scholas’s WorkAdvance program did not have a statistically sig- 
nificant effect on employment in either 2017 or 2018.!° For example, in 2018, 83 percent of 
WorkAdvance group members ever worked compared with 85 percent of control group members 
(Table 2.1). Because the program increased earnings without increasing employment, it is likely 
that WorkAdvance group members are advancing into positions with higher wages over time 


‘ONDNH data do not cover most independent contractors and, therefore, those workers would not be cap- 
tured in the employment and earnings outcomes presented in this report. As of May 2017, around 7 percent of 
workers nationally were estimated to be independent contractors (U.S. Bureau of Labor Statistics, 2018b). Some 
industries, such as IT, may have higher rates of independent contractors than others. 
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Table 2.1 


Per Scholas Impacts on Employment and Earnings 
in 2017 and 2018 


WorkAdvance Control Difference 


Group Group (Impact) P-Value 


Among the full sample 
Ever worked (%) 
2017 84.5 83.3 1.3 0.651 
2018 83.2 846 -1.4 0.624 


Total earnings ($) 
2017 32,552 28,049 4,503 ** 0.027 
2018 38,404 32,122 6,281 *** 0.004 


Earned $30,000 or more (%) 


2017 49.9 44.0 5.9 0.119 
2018 57.0 50.7 6.3 * 0.099 
Full site sample size 349 341 


Among the early cohort 
Ever worked (%) 
2017 86.3 81.0 5.3 0.173 
2018 85.0 83.5 1.5 0.696 


Total earnings ($) 
2017 33,961 26,799 7,162 ** 0.011 
2018 39,231 31,904 7,327 ** 0.018 


Earned $30,000 or more (%) 


2017 51.3 42.7 8.6 0.102 
2018 56.6 50.3 6.3 0.224 
(continued) 


(and not just working more), one of the main goals of the WorkAdvance model.!! There is further 
evidence of this in the program’s impact on a measure of high earnings: In 2018, 57 percent of 


"IT ooking at the difference in earnings across research groups among those employed also provides more 
evidence that the overall earnings impacts resulted from higher wages and not just higher rates of employment 
(this comparison is nonexperimental). Average earnings among employed individuals can be calculated by div- 
ing the average earnings for all individuals by the employment rate. In 2018, employed WorkAdvance group 
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Table 2.1 (continued) 


WorkAdvance Control Difference 


Group Group (Impact) P-Value 


Among the late cohort 
Ever worked (%) 
2017 82.4 86.0 -3.6 0.389 
2018 81.1 86.1 -5.1 0.232 


Total earnings ($) 
2017 30,487 29,941 546 0.857 
2018 37,145 32,668 4,478 0.152 


Earned $30,000 or more (%) 


2017 47.6 46.0 1.6 0.774 
2018 57.0 51.8 5.2 0.360 
Early cohort sample size 189 185 
Late cohort sample size 160 156 


SOURCE: MDRC calculations from National Directory of New Hires data. 


NOTES: Statistical significance levels are indicated as follows: *** = 
percent; ** = 5 percent; * = 10 percent. 

The differences in impacts between cohort subgroups are not statistically 
significant. 

Rounding may cause slight discrepancies in sums and differences. 


WorkAdvance group members earned $30,000 or more, a statistically significant 6 percentage 
point increase over the control group average. The difference in the likelihood of having earnings 
of $30,000 or more in 2017 is not statistically significant. 


Consistent with earlier findings, the cohort differences at Per Scholas remain large (espe- 
cially in 2017), although the differences in impacts across cohorts are not statistically significant. ! 
Per Scholas increased earnings by over $7,000 among the early cohort in 2017 and in 2018. The 
earnings differences among the late cohort were smaller and not statistically significant in either 
year. These findings contrast with initial expectations that the WorkAdvance programs would 
improve over time and, therefore, have stronger effects among the late cohort compared with the 
early cohort. The previous findings based on state unemployment insurance wage data also 
showed larger effects among the early cohort compared with the late cohort. At that time, it 
seemed like the pattern was due, at least in part, to the economy in New York City rebounding 
from the Great Recession during the study period. This hypothesis was based on the higher 


members earned $46,158 ($38,404 / 0.832), on average, compared with average earnings of $37,969 for em- 
ployed control group members ($32,122 / 0.846). 


See Hendra et al. (2016) and Schaberg (2017). 
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earnings levels for the control group in the late cohort compared with the earnings levels for those 
in the early cohort as seen in state unemployment insurance wage data. Earnings levels for the 
WorkAdvance group were similar across cohorts. This pattern of findings does not hold as 
strongly in the NDNH-based outcomes shown in Table 2.1, providing less evidence to support 
this hypothesis. ° 


St. Nicks Alliance 


St. Nicks Alliance, a large, community-based organization in New York City, offers a range of 
services including workforce programs. The organization has operated a job training program in 
the site’s targeted sector, environmental remediation, since 2001.'4 St. Nicks Alliance had diffi- 
culty fully implementing several aspects of the WorkAdvance model; it eventually added addi- 
tional training in hazardous materials transportation and pest control in the face of decreased de- 
mand for environmental remediation technicians. Box 2.4 gives an overview of St. Nicks 
Alliance’s program as of 2019, and highlights adaptations the organization has made to its train- 
ing program since the WorkAdvance evaluation ended. 


The WorkAdvance program at St. Nicks Alliance did not have a statistically significant 
effect on employment or average earnings in either 2017 or 2018 (Table 2.2). In 2018, 78 percent 
of WorkAdvance group members and 80 percent of control group members worked at some 
point. WorkAdvance group members earned an average of $26,670 that year compared with av- 
erage earnings of $23,822 for the control group. The associated $2,849 — or 12 percent — in- 
crease in earnings is not statistically significant.'> However, St. Nicks Alliance did increase the 
likelihood of individuals having earnings of at least $30,000 in 2018: That year, 41 percent of 
WorkAdvance group members and 32 percent of control group members earned $30,000 or more. 
This suggests that some WorkAdvance group members were advancing into higher-paying jobs. 


The earnings differences presented in this report are larger than those seen in previous 
findings based on the state unemployment insurance wage data (see Appendix Table A.3). How- 
ever, the sample size at St. Nicks Alliance was smaller than the sample sizes at the other three 
sites. Therefore, the impacts need to be larger at St. Nicks Alliance, relative to the other sites, in 


Previous WorkAdvance reports also explored several other possibilities for what might be driving this 
pattern of findings. An analysis of baseline data showed that there were some differences in participant charac- 
teristics across the cohorts, but changes in the composition of the sample did not appear to drive the pattern of 
outcomes. Additionally, an analysis of the participation data found that, if anything, training completion and 
credentialing differences were larger in the late cohort than in the early cohort. 

'4Jobs in the environmental remediation sector deal with the removal of pollutants and contaminants from 
the environment, including from water and soil. 

'S At least some of the difference in earnings appears to have been driven by WorkAdvance group members 
having higher wages than control group members, and not just working at a higher rate. Average earnings among 
employed individuals can be calculated by dividing the average earnings for all individuals by the employment 
rate. In 2018, employed WorkAdvance group members earned $34,324 ($26,670 / 0.777, on average, compared 
with average earnings of $29,889 for the control group ($23,822 / 0.797). This measure is nonexperimental. 
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Box 2.4 


Where Are the WorkAdvance Providers Now? 
St. Nicks Alliance 


After the WorkAdvance evaluation ended, St. Nicks Alliance resized its environmental remedi- 
ation technician (ERT) training program to match the demand it was seeing from employers and 
participants and to coincide with employers’ hiring cycles. Building on its experience, and with 
the support of two local developers as anchor employers, St. Nicks Alliance also developed a 
construction training program. An initial labor market study, which included employer and 
stakeholder interviews, indicated an overlap between the ERT training and the new construction 
training and showed that construction employers were looking for both credentialed ERT and 
construction laborer trainees. Building on its experience with the anchor employers, St. Nicks 
Alliance now works with over 100 local and regional construction firms who value the local, 
reliable, credentialed program graduates the organization provides. Since February 2016, St. 
Nicks Alliance has trained 262 participants in 17 cycles of construction training and achieved a 
90 percent placement rate. Through its strong employer relationships, the organization has also 
been able to negotiate better jobs with higher wages and more benefits for its graduates. 


St. Nicks Alliance has further adapted its training model to other sectors, including financial 
services, urban landscaping, information technology, and health care. The programs were all 
developed through partnerships with employers. For example, St. Nicks Alliance partnered with 
Bank of America to offer training in bilingual financial services and customer service. Bank of 
America staff members developed the curriculum, are actively involved in the training, and teach 
over half of the classes, and the bank has committed to hiring half of the training graduates. 


In June 2016, St. Nicks Alliance developed a business council that brings together employers on 
a quarterly basis. They discuss the current state and needs of the sectors they work in and how 
the trainings offered by the organization could be adjusted, and provide suggestions to St. Nicks 
Alliance staff members on how to strengthen the program. The organization adds new members 
and sectors to the business council as its programming expands. One idea that came out of these 
meetings and has since been adopted by St. Nicks Alliance is a young adult mentorship program. 


Building on the experience of WorkAdvance, St. Nicks Alliance has continued to strengthen its 
alumni services. Program graduates come back to the organization quarterly to network with 
one another, learn about next steps they can take in their careers, obtain industry recertification, 
and connect with St. Nicks Alliance’s employer partners. St. Nicks Alliance is also exploring 
whether to offer advanced training opportunities, after input from the business council and em- 
ployer partners. 


order to be statistically significant (see the minimum detectable effects calculated in the 
power analysis done as part of the original study in Appendix Table A.1). 

The findings for the early and late cohorts tell a similar story: The earnings differences 
among both cohorts are positive but not statistically significant in 2017 and 2018 (Table 2.2). For 
example, among the late cohort, WorkAdvance group members earned an average of $28,607 in 
2018 compared with average earnings of $26,333 for control group members. Again, it is hard to 
detect statistically significant impacts with the sample sizes of the early and late cohorts at St. 
Nicks Alliance. 
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Table 2.2 


St. Nicks Alliance Impacts on Employment and Earnings 
in 2017 and 2018 


WorkAdvance Control Difference 


Group Group (Impact) P-Value 


Among the full sample 
Ever worked (%) 
2017 79.8 81.4 -1.5 0.673 
2018 77.7 79.7 = --2.0 0.601 


Total earnings ($) 
2017 24,543 22,586 1,957 0.386 
2018 26,670 23,822 2,849 0.208 


Earned $30,000 or more (%) 


2017 36.0 29.1 6.9 0.103 
2018 41.4 32.0 9:3) =" 0.033 
Full site sample size 242 237 


Among the early cohort 
Ever worked (%) 
2017 72.9 78.9 -6.1 0.255 
2018 74.0 79.4  -5.3 0.308 


Total earnings ($) 
2017 23,555 21,791 1,764 0.589 
2018 24,430 22,262 2,168 0.483 


Earned $30,000 or more (%) 
2017 37.2 29.6 7.6 0.208 
2018 39.1 27.7 11.4 * 0.055 


(continued) 


Madison Strategies Group 


Madison Strategies Group is a nonprofit organization that provides workforce development ser- 
vices in Tulsa, Oklahoma.'* Its WorkAdvance program initially targeted the transportation sector, 
but the provider later added a focus on the manufacturing sector after it became clear that someone 


‘6Madison Strategies Group is now often known as Tulsa Community Work Advance. 
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Table 2.2 (continued) 


WorkAdvance Control Difference 


Group Group (Impact) P-Value 


Among the late cohort 
Ever worked (%) 
2017 87.1 84.8 2.3 0.634 
2018 80.9 81.1 -0.1 0.983 


Total earnings ($) 
2017 25,210 24,028 1,181 0.714 
2018 28,607 26,333 2,274 0.509 


Earned $30,000 or more (%) 


2017 33.3 29.9 3.5 0.570 
2018 42.9 38.4 4.5 0.500 
Early cohort sample size 127 131 
Late cohort sample size 115 106 


SOURCE: MDRC calculations from National Directory of New Hires data. 


NOTES: Statistical significance levels are indicated as follows: *** = 
percent; ** = 5 percent; * = 10 percent. 

The differences in impacts between cohort subgroups are not statistically 
significant. 

Rounding may cause slight discrepancies in sums and differences. 


who is trained to manufacture transportation-related parts has the requisite skills to work in man- 
ufacturing more generally. The organization was new to Tulsa at the beginning of the study,!’ and 
it took some time for it to establish relationships with training providers and employers and to 
fully implement the WorkAdvance model components. (Box 2.5 describes Madison Strategies 
Group’s program as of 2019). This initial start-up period, as well as the organization’s use of the 
placement-first track in the early part of the study period, meant that early enrollees probably 
experienced a less mature program than late enrollees. For these reasons, the late cohort findings 
may better represent the effects of Madison Strategies Group’s WorkAdvance program. 


Madison Strategies Group did not increase employment by a statistically significant 
amount among the late cohort in 2017 or 2018 (Table 2.3). In 2018, 74 percent of WorkAdvance 
group members were ever employed compared with 79 percent of control group members. The 
impacts on average earnings among the late cohort are not statistically significant in either year. 


'’Madison Strategies Group is a nonprofit spinoff of Grant Associates, a for-profit workforce development 
company with sector program experience in New York City. Madison Strategies Group was able to take some 
institutional knowledge from its parent organization. 
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Box 2.5 
Where Are the WorkAdvance Providers Now? 


Madison Strategies Group 


When the WorkAdvance evaluation started in 2011, Madison Strategies Group was a new or- 
ganization in Tulsa, Oklahoma, and had to ramp up its services quickly. Since the evaluation 
ended, the organization has continued to offer services based on the WorkAdvance model, but 
it has focused its efforts on figuring out how to adapt those services to make the program even 
stronger. The organization now has a much larger presence in its community, and based on its 
achievements, has become a leader in the local workforce field. 


Madison Strategies Group continues to operate training programs in the manufacturing and 
transportation sectors, but it has changed some of the specific trainings it offers based on the 
types of jobs available to graduates. For example, the organization now offers training to obtain 
a Commercial Driver’s License Class B (instead of a Commercial Driver’s License Class A) 
because there are more opportunities to obtain good jobs with higher wages. Madison Strategies 
Group also offers training in other sectors that have more economic mobility, including account- 
ing, information technology, logistics, and health care. 


Madison Strategies Group has also focused on its recruitment and intake process. The organiza- 
tion overhauled its interview questions, developed a point scale, and found new assessments to 
use for each training track. These adaptations have helped staffmembers to better identify which 
applicants will benefit the most from the program services. 


Madison Strategies Group has also launched two new initiatives that build on the success of the 
WorkAdvance model. First, the organization is piloting a public housing-based program that is 
funded through a Choice Neighborhood grant. This program draws on aspects of the WorkAd- 
vance model to deliver job readiness and employment services, including workshops and indi- 
vidualized coaching, to individuals living in a specific neighborhood. The services are designed 
to increase job skills and employment. Additionally, the organization has started a program 
aimed at 18- to 24-year-olds called “NextUp.” Participants receive a set of support services over 
several months in addition to their occupational skills traning. They work with a coach on per- 
sonal issues such as how to set up a bank account, and with a career advisor on professional 
issues such as how to get through the occupational skills training. 


In 2018, WorkAdvance group members in the late cohort earned $23,616, on average, compared 
with average earnings of $22,060 for the control group. Among the early cohort, the earnings 
differences are also positive, though smaller, and statistically insignificant (the differences in im- 
pacts across cohorts are not statistically significant). However, Madison Strategies Group did in- 
crease the likelihood of having high earnings ($30,000 or more) among the late cohort in 2017. 
That year, 38 percent of WorkAdvance group members earned at least $30,000, a statistically 
significant increase of 12 percentage points over the control group average. This suggests that 
some WorkAdvance group members were advancing into higher-paying jobs that year. 
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Table 2.3 


Madison Strategies Group Impacts on Employment 
and Earnings in 2017 and 2018 


WorkAdvance Control Difference 


Group Group (Impact) P-Value 


Among the full sample 
Ever worked (%) 
2017 77.2 78.6 = -1.3 0.663 
2018 74.1 78.6 8 -4.4 0.168 


Total earnings ($) 
2017 19,739 17,958 1,780 0.177 
2018 21,248 20,461 787 0.603 


Earned $30,000 or more (%) 


2017 32.5 21.0 11.5 *** 0.001 
2018 33.6 27.8 5.8 * 0.092 
Full site sample size 353 344 


Among the early cohort 
Ever worked (%) 
2017 71.9 76.0 = -4.1 0.394 
2018 71.3 73.6 -2.4 0.626 


Total earnings ($) 
2017 17,855 15,873 1,983 0.287 
2018 18,869 18,614 254 0.903 


Earned $30,000 or more (%) 
2017 27.7 16.0 11.7 *** 0.008 
2018 27.6 24.0 3.6 0.449 


(continued) 


Among the full sample, WorkAdvance group members earned more, on average, than 
control group members: in 2017, $1,780 (or about 10 percent) more; and in 2018, $787 more. 
However, neither difference is statistically significant (Table 2.3). In both years, Madison Strate- 
gies Group increased the likelihood of earning at least $30,000 by a statistically significant 
amount. For example, in 2018, 34 percent of WorkAdvance group members earned $30,000 or 
more compared with 28 percent of control group members. 
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Table 2.3 (continued) 


WorkAdvance Control Difference 


Group Group (Impact) P-Value 


Among the late cohort 
Ever worked (%) 
2017 82.7 80.7 2.0 0.617 
2018 77.2 82.8 -5.5 0.201 


Total earnings ($) 
2017 21,713 19,695 2,018 0.283 
2018 23,616 22,060 1,556 0.482 


Earned $30,000 or more (%) 


2017 37.5 25.3 12.2 ** 0.011 
2018 39.4 31.2 8.2 0.105 
Early cohort sample size 173 164 
Late cohort sample size 180 180 


SOURCE: MDRC calculations from National Directory of New Hires data. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** 
= 5 percent; * = 10 percent. 

The differences in impacts between cohort subgroups are not statistically 
significant. 

Rounding may cause slight discrepancies in sums and differences. 


There were no statistically significant differences in employment rates in either year 
among the full sample. In 2017, 77 percent of WorkAdvance group members and 79 percent of 
control group members were employed at some point. Furthermore, 32 percent of WorkAdvance 
group members and 37 percent of control group members were hired at a new job that year (see 
Appendix Table A.7), indicating a high rate of job change among both research groups. '* It should 
be noted that previous findings from the survey data showed Madison Strategies Group’s program 
had positive impacts on several work-related measures — for example, the availability of em- 
ployer-provided benefits — that cannot be measured in the administrative records data. It is pos- 
sible that these impacts persisted in the long term. 


Towards Employment 


Towards Employment is an established community-based organization in northeast Ohio that 
provides a range of employment services. It targeted both the health care and manufacturing sec- 
tors. Before it implemented the WorkAdvance model, the organization focused more on work 


'8These job changes could be related to sample members advancing into better jobs or to sample members 
losing their jobs and needing to find new ones. 
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readiness than on technical training but had experience with programs targeted at entry-level jobs 
in the health care sector. The evaluation required Towards Employment to add career advance- 
ment services, deepen its expertise within the health care sector, and branch out and develop re- 
lationships with new training providers and employers within a new sector, manufacturing.'? To- 
wards Employment adjusted the specific training and credentials it offered in both targeted sectors 
throughout the study period based on employer needs. (See Box 2.6 for an update on changes 
Towards Employment made to its program after the evaluation period.) Given these changes, as 
well as the organization’s use of the placement-first track early on, it was hypothesized that the 
effects of WorkAdvance would be stronger for the late cohort than for the early cohort. 


WorkAdvance group members and control group members in the late cohort worked and 
earned similar amounts in 2017 and 2018 (Table 2.4). For example, in 2018, WorkAdvance group 
members earned $22,100, on average, and control group members earned $22,086, on average.” 
Among the early cohort, the earnings differences are larger but not statistically significant. The 
estimated differences in impacts across the cohort subgroups are not statistically significant. This 
finding goes against the hypothesis that the effects would be stronger for the late cohort than for 
the early cohort, as well as previous findings based on the state unemployment insurance wage 
data that showed statistically significant earnings effects among the late cohort in some years (see 
Appendix Table A.5). Among the late cohort, there is a statistically significant effect on the like- 
lihood of having earnings of $30,000 or more in 2017: That year, 33 percent of WorkAdvance 
group members earned at least that much compared with 25 percent of control group members. 


Towards Employment did not have a statistically significant effect on employment or 
earnings in 2017 or 2018 among the full sample (Table 2.4). In 2018, WorkAdvance group mem- 
bers earned around $1,400 — or 8 percent — more, on average, than control group members. 
That year, 79 percent of both WorkAdvance group members and control group members were 
employed.*! Previous findings from the Year 2 survey data showed Towards Employment’s 
WorkAdvance program had positive impacts on some work-related measures, including work 
schedules and job types, that cannot be measured in the administrative records data. It is possible 
that these impacts persisted in the long term. 


'°Two other implementation factors were unique to Towards Employment’s program: (1) The organization 
initially oversaw a second program location that was ultimately not included in the analysis, and (2) it managed 
partnerships with other service providers, educational institutions, trade organizations, and labor market inter- 
mediaries that delivered various components of the model, an arrangement that is probably typical of many sector 
programs. See Tessler et al. (2014) and Hendra et al. (2016) for more details. 

°Towards Employment’s WorkAdvance program increased average earnings in 2017 by a statistically sig- 
nificant amount among the late cohort based on state unemployment insurance wage data (Appendix Table A.5). 

*1 Appendix Table A.6 shows impacts on employment and earnings by random assignment sector at To- 
wards Employment. These impacts are based on state unemployment insurance wage data. Among sample mem- 
bers randomly assigned in the manufacturing sector, WorkAdvance increased employment in Year 2 by a statis- 
tically significant amount. In Year 4, the estimated impact on earnings is statistically significant among sample 
members randomly assigned in the health care sector. 
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Box 2.6 
Where Are the WorkAdvance Providers Now? 


Towards Employment 


Since the original WorkAdvance evaluation ended, Towards Employment has adapted its man- 
ufacturing and health care trainings in response to shifts in the local labor market. With lower 
unemployment rates, staff members say employers are more willing to partner with them to 
address current skills gaps. Staff members also say job seekers are less likely to go to training, 
choosing to enter the labor market instead. In response, Towards Employment has started work- 
ing more directly with employers on “earn and learn” opportunities for new and current employ- 
ees and is exploring how to use incentives to get more individuals into its trainings. 


In the health care sector, the organization has continued to grow its partnership with University 
Hospitals (UH). Towards Employment refers program graduates to open positions in the hospi- 
tal system, provides retention and advancement services to those who were hired, and has intro- 
duced internal “pathways programs” to help current employees advance to higher-level posi- 
tions. The strong relationship with UH has allowed Towards Employment to be nimble — 
quickly adapting its focus and services as the employer’s needs change. As of fall 2019, Towards 
Employment had placed more than 350 individuals in jobs with UH, has helped more than 80 
individuals already working there advance into higher wage positions, and has seen a 90 percent 
one-year retention rate. 


Towards Employment has expanded and refined its programming to work with new populations 
— for example, individuals being released from prison — and to reach other industries, such as 
the construction and culinary trades. One innovative approach adopted by the organization: To- 
wards Employment operated its own social enterprise bakery and café for four years to provide 
transitional jobs and training for individuals in its culinary training program. 


Towards Employment has also placed an increased emphasis on long-term, advancement-fo- 
cused career coaching in its program model. The organization offers two years of career path- 
ways support to participants, including one year after participants have found jobs. Participants 
work with a coach to map out a career plan that includes goals and the strategies to achieve them. 
Towards Employment has also continued to work with people who were part of the original 
WorkAdvance evaluation. Many of these individuals continue to reach out to the organization 
to share their successes and to ask for guidance on how they can advance to the next stage of 
their careers. 


Pooled Sample 


While the main WorkAdvance impact analysis was done at the site level, given the substantial 
variation among the providers, it is useful to understand how a model like WorkAdvance might 
perform overall, across providers. Table 2.5 shows the impacts of WorkAdvance for the pooled 
sample, combining sample members from all four WorkAdvance sites. 


WorkAdvance had no effect on employment in either 2017 or 2018. In 2018, 79 percent 
of WorkAdvance group members and 80 percent of control group members were employed at 
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Table 2.4 


Towards Employment Impacts on Employment and Earnings 
in 2017 and 2018 


WorkAdvance Control Difference 


Group Group (Impact) P-Value 
Among the full sample 
Ever worked (%) 
2017 82.5 80.8 1.7 0.541 
2018 79.0 78.6 0.4 0.894 
Total earnings ($) 
2017 19,006 17,477 1,529 0.197 
2018 19,742 18,338 1,404 0.275 
Earned $30,000 or more (%) 
2017 26.7 21.5 5.2 0.100 
2018 28.9 23.8 5.1 0.117 
Full site sample size 349 349 
Among the early cohort 
Ever worked (%) 
2017 77.8 75.4 2.4 0.588 
2018 75.6 72.8 2.8 0.541 
Total earnings ($) 
2017 16,461 14,182 2,279 0.148 
2018 17,172 14,726 2,446 0.139 
Earned $30,000 or more (%) 
2017 20.1 18.2 1.9 0.650 
2018 22.0 18.7 3.3 0.445 
(continued) 


some point. However, WorkAdvance did increase earnings by statistically significant amounts in 
both years: by $2,392 in 2017 and by $2,716 in 2018. Because there are earnings increases with- 
out commensurate increases in employment for the pooled sample, there is evidence that Work- 
Advance did lead to some wage gains. This is a sign that WorkAdvance group members were 
advancing over time, as the model intended. Further evidence of advancement is seen in the sta- 
tistically significant impacts on the likelihood of individuals having earnings of at least $30,000 
in both 2017 and 2018. 
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Table 2.4 (continued) 


WorkAdvance Control Difference 


Group Group (Impact) P-Value 
Among the late cohort 
Ever worked (%) 
2017 86.9 86.4 0.5 0.894 
2018 81.8 84.9 -3.1 0.441 
Total earnings ($) 
2017 21,354 20,882 472 0.791 
2018 22,100 22,086 14 0.994 
Earned $30,000 or more (%) 
2017 32.8 24.8 8.1 * 0.093 
2018 35.3 29.2 6.1 0.216 
Early cohort sample size 168 177 
Late cohort sample size 181 172 


SOURCE: MDRC calculations from National Directory of New Hires data. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

The differences in impacts between cohort subgroups are not statistically 
significant. 

Rounding may cause slight discrepancies in sums and differences. 


While the earnings impacts are statistically significant and show the impact for a range 
of possible WorkAdvance providers, they do mask the variation in impacts across the sites dis- 
cussed in the previous section. In other words, some of the pooled sample earnings impacts are 
being driven by the large earnings impacts at Per Scholas. However, an exploratory analysis look- 
ing at the impacts among the pooled sample from St. Nicks Alliance, Madison Strategies Group, 
and Towards Employment showed statistically significant earnings increases in some quarters 
(not shown).” This suggests that the earnings impacts of WorkAdvance were not limited to Per 
Scholas. It should also be noted that because the site-specific sample sizes are smaller relative to 
the pooled sample size, the site-specific impacts need to be larger to be statistically significant (a 
power analysis from the original study is presented in Appendix Table A.1). 


?This analysis indicated that among the pooled samples at St. Nicks Alliance, Madison Strategies Group, 
and Towards Employment, WorkAdvance increased earnings by statistically significant amounts in Quarter 3, 
2017 through Quarter 1, 2018; in Quarter 4, 2018; and in 2017 overall. 
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Table 2.5 


Impacts on Employment and Earnings in 2017 and 2018 
for the Pooled Sample 


WorkAdvance Control Difference 
Outcome group group (Impact) P-Value 


Ever worked (%) 
2017 81.2 80.9 0.4 0.816 


2018 78.8 80.2 -1.5 0.355 


Total earnings ($) 
2017 23,844 21,452 2,392 *** 0.005 
2018 26,419 23,703 2,716 *** 0.003 


Earned $30,000 or more (%) 


2017 36.1 28.9 7.2 *** 0.000 
2018 40.0 33.7 6.4 *** 0.001 
Sample size 1,293 1,271 


SOURCE: MDRC calculations from National Directory of New Hires data. 


NOTE: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 
Rounding may cause slight discrepancies in sums and differences. 


Labor Market Attachment Subgroup 


Among the individuals who enrolled in the WorkAdvance study, there is considerable variation 
in terms of demographics, education and work experience, and other characteristics. At the start 
of the study, it was an open question whether WorkAdvance would work better for some individ- 
uals than for others and whether the program should target a more well-defined group. To help 
answer this, the second prespecified confirmatory subgroup analysis was based on sample mem- 
bers’ levels of attachment to the labor market at the time they entered the study.” The labor mar- 
ket attachment subgroup analysis was done at the pooled sample level — combining sample 
members from all four WorkAdvance providers — due to small site sample sizes. 


3In previous reports, impacts for several exploratory subgroups, including age, race, prior education, prior 
earnings, and prior conviction or incarceration status, were also analyzed using state unemployment insurance 
wage data. The overall impression from those analyses was that the impacts of WorkAdvance did not vary greatly 
across many subgroups. Some subgroups experienced larger impacts than others, but in most cases the variation 
in impacts across subgroups was not statistically significant. It is unclear whether those findings would hold for 
the NDNH impacts presented in this report. It was not possible to repeat these subgroup analyses with the long- 
term data due to restrictions on access to the NDNH data. 
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The labor market attachment subgroup analysis split the sample into three groups: (1) the 
fully attached, those who were employed or who had been out of work for less than one month at 
study entry; (2) the semiattached, those who had been out of work for between one and six months 
at study entry; and (3) the long-term unemployed, those who had never worked or who had been 
out of work for seven or more months at study entry. Based on previous studies, it was hypothe- 
sized that WorkAdvance would be most effective for individuals in the semiattached group who 
had some connection to the labor market. The rationale was that these individuals are often at a 
“tipping point” in their employment trajectories and therefore might be more sensitive to the in- 
tervention. The WorkAdvance services could help them get back into the workforce, which was 
not an issue for the fully attached group,” yet they would not have too many barriers to overcome, 
as the long-term unemployed might, in order to benefit from the program. The long-term unem- 
ployed group was also relevant to policy, given that WorkAdvance was implemented in the wake 
of the Great Recession and there was significant concern about the probability of this group reen- 
tering the labor market. 


There were no statistically significant effects on employment in any subgroup in 2017 or 
2018. The difference in employment impacts across subgroups was also not statistically signifi- 
cant. However, the difference in earnings impacts in 2018 across subgroups was statistically sig- 
nificant. In that year, WorkAdvance increased earnings among the semiattached by $4,745 and 
among the long-term unemployed by $3,235; the earnings impact among the fully attached group 
was not statistically significant (Table 2.6). Among the semiattached, there was also a large and 
statistically significant impact of more than $4,200 on earnings in 2017.” In that year, semiat- 
tached WorkAdvance group members earned $25,645, on average, compared with average earn- 
ings of $21,392 for semiattached control group members. 


As might be expected, both the employment and earnings levels were higher for both 
research groups among the semiattached group than among the long-term unemployed group. 
Nonetheless, it is encouraging to see that WorkAdvance was able to help the long-term unem- 
ployed reenter the labor market and increase their earnings, relative to what would have happened 
in the absence of the program. 


These findings on their own suggest that WorkAdvance can increase earnings for the 
semiattached and long-term unemployed groups, but not necessarily that WorkAdvance works 
better for those groups. Some of the impacts among those groups were likely driven by the site- 
specific impacts, as were the lack of impacts among the fully attached group. A similar pattern of 


4The Employment Retention and Advancement evaluation and other prior welfare-to-work studies showed 
an “inverted U” subgroup pattern. See Hendra et al. (2010). 

>The WorkAdvance model was designed to help individuals who were already employed by offering ad- 
vancement-focused services. The postemployment services envisioned by the model were to be more robust than 
the services offered to participants postplacement in previous evaluations. 

>The results are consistent with the pattern of findings seen in other studies. See, for example, Hendra et al. 
(2010). 
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Table 2.6 


Impacts on Employment and Earnings in 2017 and 2018 for Subgroups Defined by 
Baseline Labor Market Attachment, Among the Pooled Sample 


Fully attached Semiattached Long-term unemployed 
WA C Difference WA C_ Difference WA C Difference 

Outcome group group (Impact) group group _ (Impact) group group (Impact) Sig. 
Ever worked (%) 

2017 87.7 89.6 -1.9 82.4 80.7 1.6 75.8 75.2 0.6 

2018 84.9 87.5 -2.6 80.7 81.5 -0.9 73.0 74.1 -1.1 
Total earnings ($) 

2017 25,456 25,362 94 25,645 21,392 4,252 *** 20,872 19,098 1,774 

2018 26,557 27,696 -1,139 28,965 24,220 4,745 *** 23,787 20,552 3,235 ** tt 
Sample size 308 356 456 449 529 463 


SOURCE: MDRC calculations from National Directory of New Hires data. 


NOTES: WA = WorkAdvance; C = Control. 

The fully attached group consists of sample members who at baseline were working or had been unemployed for less than 
one month. The semiattached group consists of sample members who had been unemployed for one to six months at 
baseline. The long-term unemployed group consists of sample members who had never been employed or who had been 
unemployed for seven or more months at baseline. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as 
follows: ttt = 1 percent; tt = 5 percent; t = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 


findings for the labor market attachment subgroups was seen previously in the impacts based on 
state unemployment insurance wage data. An analysis of those findings showed the impacts 
among the semiattached and long-term unemployed groups were partly due to the site-specific 
impacts.*’ The findings do support the idea that sector programs are not “one size fits all.”” Some 
groups of participants may need different services than others, and programs may need to change 
their service offerings to meet the needs of individual participants. 


?7The analysis used a regression that controlled for the program site. The findings showed that the stronger 
effects among both the semiattached and the long-term unemployed were eliminated when controlling for site. 
See Hendra et al. (2016) and Schaberg (2017) for more information. It was not possible to recreate this analysis 
with the NDNH data. 
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Chapter 3 


Benefit-Cost Analysis of WorkAdvance 


This section examines whether the various effects of each of the four WorkAdvance programs, 
when viewed in aggregate, resulted in financial gains or losses. These gains and losses are re- 
ported from three perspectives: those of persons assigned to the WorkAdvance group (“partici- 
pants”); the government;! and society as a whole, which combines the first two perspectives and 
also includes any benefits and costs that accrue to members of the control group and persons not 
involved in the study (“nonparticipants’’). 


Several aspects of the benefit-cost analysis are consistent with the economic impact anal- 
ysis presented in the previous section. For example, all the findings from the benefit-cost analysis 
are reported on a per-participant basis. Moreover, all of the persons assigned to the WorkAdvance 
group, both those for whom the program had effects and those for whom it did not, were included 
in the benefits and costs estimate. 


There are also important differences between the benefit-cost study and the impact anal- 
ysis. For instance, unlike the impact analysis, the benefits and costs for Madison Strategies Group 
and Towards Employment are based on only the late cohort, while those for Per Scholas and St. 
Nicks Alliance are based on the full sample. Thus, persons in the first two sites who might have 
entered the “placement first” track, rather than the “training first” track, are excluded from the 
analysis and the results are therefore applicable to a “purer” version of WorkAdvance. In addition, 
the benefit-cost study takes a broader look at WorkAdvance than the impact study — for example, 
by incorporating the costs of running the programs and the programs’ effects on taxes, transfer 
payments, and the receipt of fringe benefits. In addition, the benefit-cost study uses estimates of 
benefits and costs regardless of their level of statistical significance. However, after reporting the 
findings from the study, statistical significance and other sources of uncertainty concerning them 
are examined through sensitivity analyses. 


Conducting the Benefit-Cost Study 


To conduct the benefit-cost analyses, it was first necessary to determine the costs of operating 
each of the four WorkAdvance programs. A cost study to do this was conducted previously and 
the findings are summarized earlier in this report. The starting point for the cost study was finan- 
cial reports that the sites were required to submit over the life of the program. Details about how 
the cost study was done and additional findings from it can be found in Chapter 4 of the report on 
WorkAdvance’s two-year impacts.* 


‘Much of the cost of operating the WorkAdvance demonstration was paid for by foundations. Nonetheless, 
the benefit-cost analysis treats WorkAdvance under the assumption that it was entirely funded by the government 
as likely would be the case were the program to became permanent. 


"Hendra et al. (2016). 
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Appendix B provides details on how each of the remaining program benefits and costs 
were estimated. As indicated there, the earnings impacts used in the benefit-cost analysis are 
based on both state unemployment insurance wage data and National Directory of New Hires 
data. WorkAdvance’s estimated impacts on earnings feed into the measurement of many of the 
other benefits and costs used in the analysis. For example, impacts on earnings engender changes 
in fringe benefits, tax payment amounts, work-related expenditures, and time available outside of 
work (sometimes called nonmarket or leisure time, which is of value to most people). In addition, 
paying for WorkAdvance requires additional taxes. Reductions in transfer payments received by 
program participants, on the other hand, such as food stamps/Supplemental Nutrition Assistance 
Program (SNAP) and housing assistance, as well as increases in tax payments paid by partici- 
pants, means that taxes paid by others can be reduced. These changes in tax receipts and transfer 
payments, in turn, increase or decrease inefficiencies in the economy (for example, by influencing 
decisions affecting how much to work or invest), inefficiencies that economists call “deadweight 
loss.” As explained in Appendix B, impacts on these items are estimated by using information 
from various external sources to determine how these benefits and costs change as earnings 
change. For example, an increase in fringe benefits attributable to a dollar increase in earnings is 
derived from an externally obtained multiplier that incorporates data collected by the U.S. Bureau 
of Labor Statistics, as well as data collected in the WorkAdvance Year 2 survey. 


As many benefits and costs as possible were estimated in dollars. Summing these items 
indicates whether the financial gains due to WorkAdvance exceeded the financial losses. How- 
ever, it was not possible to measure all the possible benefits and costs resulting from WorkAd- 
vance in dollars. For example, by working more, participants in WorkAdvance may have filled 
job openings that otherwise would have been taken by nonparticipants. Also, as a result of par- 
ticipants receiving increased earnings, their spouses and partners may have been able to work 
less. Appendix C considers half a dozen potential benefits and costs that could not be measured 
in dollars and whether, if they could be measured, they would substantially affect the findings 
based on the benefits and costs that are estimated in dollars, concluding that they probably would 
not. 


Some of the benefits and costs considered in this section do not directly affect society as 
a whole. For example, an increase of a dollar in income tax payments paid by participants or a 
dollar reduction in SNAP benefits received by participants is offset by a dollar improvement in 
the government’s fiscal position. However, by reducing the taxes that persons who did not par- 
ticipate in WorkAdvance must pay, this improvement results in a reduction in deadweight loss, 
which does affect society as a whole because it is not offset by a cost occurring elsewhere. More- 
over, a reduction in SNAP benefits that reduces SNAP’s administrative cost is not offset by a cost 
to participants and, hence, benefits society as a whole. 


The effects of WorkAdvance are unlikely to suddenly end when the period for which 
earnings data were collected ends. MDRC projected benefits and costs until the tenth year after 
random assignment or until it appeared that program impacts on earnings fell to zero, whichever 
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occurred first.? Thus, it is assumed that once WorkAdvance increased earnings, it would not sub- 
sequently reduce earnings. As discussed in Appendix B, it appears that positive impacts on earn- 
ings were likely to continue at St. Nicks Alliance for more than ten years but to disappear after 
five and a half years at Madison Strategies Group and Towards Employment and after seven years 
at Per Scholas. 


Because WorkAdvance’s benefits and costs were received or incurred in different years, 
and those accruing later are of less value than those accruing earlier, a discount rate of 3.5 percent 
was used to convert all benefits and costs to their value in 2018.* This allows monetary values 
accrued in different years to be appropriately compared and summed. To further this comparison, 
MDRC also adjusted all the monetary estimates used in the analysis to 2018 prices using the 
Consumer Price Index (CPI). 


Findings from the Analysis of Benefits and Costs 


Like the impact estimates, the findings from the benefit-cost analysis are reported separately for 
each site. As will be seen, they are generally positive and substantial for all the sites; indeed, they 
are considerably larger than is typically the case for employment and training programs. For ex- 
ample, the net gains for society as a whole were almost $6,000 at Towards Employment, more 
than $13,000 each at St. Nicks Alliance and Madison Strategies Group, and $31,000 at Per Scho- 
las. The net gains were also large for participants, ranging from over $5,000 to over $15,000, with 
Per Scholas again at the top end of the range. Although it paid substantially to operate WorkAd- 
vance, the government did no worse than break even at one site (Towards Employment) and 
received positive returns on its investment at the other three sites, especially at Per Scholas.° It 
appears unlikely that these gains would be strongly affected if those benefits and costs that could 
not be measured in dollars could be included in the net gain estimates. 


>The period over which earnings data were collected is called “the observation period” and the period over 
which earnings are projected is called “the projection period.” As indicated above, determining whether impacts 
on earnings are positive during the observation period depends on the values of the impact estimates, not on 
whether those estimates are statistically significant. 

‘The need for a discount rate reflects the idea that most people would rather have a sum of money available 
today than have to wait several years for the same sum. There is considerable uncertainty and controversy over 
the appropriate discount rate to use in benefit-cost studies of social programs. Boardman, Greenberg, Vining, 
and Weimer (2018) suggest 3.5 percent, with 2.0 percent and 5.5 percent used to conduct sensitivity analyses. 
The estimated net gains for WorkAdvance that are reported below are relatively insensitive to these alternative 
rates, typically changing by less than 10 percent. This is because these net gains are computed over a rather short 
period of no more than 10 years. 

5As will be discussed later in this chapter, results from a Monte Carlo analysis suggest that it is difficult to 
know for certain whether there were net losses or net gains from the government’s perspective at St. Nicks Alli- 
ance, Madison Strategies Group, and Towards Employment. Nonetheless, whatever the direction, they were 
probably small. 
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Per Scholas 


Table 3.1 presents findings from the benefit-cost analysis of Per Scholas. The table re- 
ports benefits and costs from the three perspectives mentioned above: WorkAdvance participants, 
the government’s budget, and society as a whole. Benefits have positive signs in the table and 
costs have negative signs. Because they are more speculative than the other individual estimates 
(see Appendix B), the estimates of the cost of the loss of time outside of work and the gains 
resulting from reductions in deadweight loss are excluded from the top panel of the table, but they 
are discussed below and are incorporated into the net financial gain estimates appearing in the 
bottom panel. In addition, the table is limited to the period during which the data used in the 
analysis were actually collected (that is, the observation period), which lasted for five and three- 
quarters years at Per Scholas. Results projected beyond the observation period, which are more 
speculative than those limited to the observation period, are discussed below and are also reported 
in Table 3.2. 


As Table 3.1 shows, the key benefit from the participants’ perspective is their gain in 
pretax earnings. Because participants are part of society, pretax earnings are also the key benefit 
from the societal perspective. WorkAdvance participants enjoyed a large increase in earnings, so 
their fringe benefits also increased. These gains for participants and society were partially offset 
by work-related expenditures (mostly transportation). In addition, participants had to pay more 
income, payroll, and sales taxes. These increases in tax payments by participants accrue dollar- 
for-dollar to the government’s budget. Thus, they do not affect society as a whole. As might be 
expected, most transfer payments (that is, Temporary Assistance for Needy Families, SNAP, 
housing assistance, and unemployment insurance) received by WorkAdvance participants at Per 
Scholas declined. The exception was Supplemental Security Income/Social Security Disability 
Insurance payments. Like tax payments, the changes in transfer payments were offset dollar-for- 
dollar by changes in the government’s budget. Hence, there was again no effect on society as a 
whole. There was a small decrease in the cost of administering transfer benefits, however, which 
benefited both the government and society. As shown in Table 3.1, the most important cost to the 
government was the expense of operating WorkAdvance. 


When the individual benefits and costs listed in Table 3.1 are summed but nonmarket 
time and deadweight loss are not included, they indicate that the total gains from WorkAdvance 
were positive and quite substantial from all three perspectives at Per Scholas. Because WorkAd- 
vance improved the government’s budgetary position and therefore, fewer taxes had to be col- 
lected from persons who did not participate in the program, the program reduced inefficiencies in 
the economy. Thus, deadweight loss is estimated to fall by $2,167. This reduction only affects net 
gains from the societal perspective because the effect results from reductions in taxes paid by 
nonparticipants. Because the earnings increases enjoyed by program participants resulted, in part, 
from increased hours of work, participants had fewer hours available outside of work. The value 
of this loss is estimated to be $4,539. This loss would have been even larger, but, as suggested by 
the earnings threshold measure discussed earlier and by an analysis in Appendix B, much of the 
earnings increase is due to advancement and higher wage rates, rather than more time spent at 
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Table 3.1 


Benefits and Costs for Per Scholas by Accounting Perspective 
Over the 63-Month Observation Period (in 2018 dollars) 


Government 

Benefits and costs ($) Participants Budget Society 
Pretax earnings of participants 28,661 0 28,661 
Fringe benefits 7,159 0 7,159 
Payroll taxes -4,707 4,707 0 
Income taxes -4,928 4,928 0 
Sales taxes -840 840 0 
Work-related expenditures -2,163 0 -2,163 
TANF -1,291 1,291 0 
SNAP -2,500 2,500 0 
SSI/SSDI 1,177 -1,177 0 
Housing assistance -1,677 1,677 0 
UI -355 355 0 
Administrative cost of transfer programs 0 709 709 
Net program costs 0 -4,459 -4,459 
Net financial gains or losses without 

nonmarket time and deadweight loss 18,536 11,370 29,906 
Nonmarket time -4,539 0 -4,539 
Deadweight loss 0 0 2,167 
Net financial gains or losses with 

nonmarket time and deadweight loss 13,997 11,370 27,535 


SOURCES: The sources and derivation of net program costs are described in Hendra et al. 
(2016). The sources and derivation of the remaining benefit and cost components are described 
in Appendix B. 


NOTES: All benefits and costs are in dollars per participant over five years and are inflation- 
adjusted to 2018 dollars and discounted to 2018 present values. 

Rounding may cause slight discrepancies in calculating sums. 

TANF = Temporary Assistance for Needy Families, SNAP = Supplemental Nutrition 
Assistance Program, SSI = Supplemental Security Income, SSDI = Social Security Disability 
Insurance, UI = Unemployment Insurance 

Rounding may cause slight discrepancies in sums and differences. 


work. Taken together, the estimates of changes in deadweight loss and time outside of work re- 
duce estimated net gains from the participant perspective from $18,536 to $13,997 and net gains 
from the societal perspective from $29,906 to $27,535. Net gains from the perspective of the 
government’s budget are unaffected. 
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Table 3.2 


Benefits and Costs by Accounting Perspective Over 
the Observation Period and for Two Sites Over the 
Observation and Projection Periods (in 2018 Dollars), by Site 


Government 

Net financial gains and losses ($) Participants Budget Society 
Per Scholas 

Over the observation period 13,997 11,370 27,535 

Over the observation and projection periods 15,456 13,387 31,387 
St. Nicks Alliance 

Over the observation period 1,623 -4,130 -3,293 

Over the observation and projection periods 9,387 3,660 13,742 
Madison Strategies Group 

Over the observation period 11,192 1,615 13,114 

Over the observation and projection periods 11,192 1,615 13,114 
Towards Employment 

Over the observation period 5,505 265 5,820 

Over the observation and projection periods 5,505 265 5,820 


SOURCES: The sources and derivation of net program costs are described in Hendra et al. 
(2016). The sources and derivation of the remaining benefit and cost components are described 
in Appendix B. 


NOTES: The observation period for Per Scholas and St. Nicks Alliance is 63 months and for 
Madison Strategies Group and Towards Employment is 62 months. The combined observation 
and projection period for Per Scholas is 7 years and for St. Nicks Alliance is 10 years. For 
Madison Strategies Group and Towards Employment, the benefits and costs were not projected 
beyond the observation period because it appears the positive impacts on earnings 
disappeared after the observation period. 

All gains and losses include the monetized effects of WorkAdvance on nonmarket time and 
deadweight loss, are inflation-adjusted to 2018 dollars, and are discounted to 2018 present 
values. 


Using the approach described in Appendix B, it was estimated that WorkAdvance at Per 
Scholas produced positive effects on earnings for a total of seven years (that is, for five quarters 
beyond the observation period). Taking account of these additional five quarters, and also of the 
loss of nonwork time and improvement in deadweight loss, it is projected that the net gains at Per 
Scholas were $15,456 for participants, $13,387 from the government budgetary perspective, and 
$31,387 for society as a whole (see Table 3.2). These net gains are larger than those engendered 
by the other three WorkAdvance programs examined in this report. Indeed, as discussed later, 
they are exceptionally large as compared with those resulting from most employment and training 
programs. 
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As mentioned above, halfa dozen benefits and costs that could not be estimated in dollars 
are considered in Appendix C. Three of them appear, if anything, potentially likely to add to the 
substantial gains reported in Table 3.1, though very modestly. Two potential effects of WorkAd- 
vance at Per Scholas that could not be measured would probably reduce the net gains appearing 
in Table 3.1 if they could be estimated in dollars. However, for reasons discussed in Appendix C, 
the potential size of the first of these effects — the possibility that because the earnings of partic- 
ipants were larger, their spouses and partners worked less — 1s probably very modest. The second 
effect — the possibility that competition from WorkAdvance participants causes nonparticipants 
to earn less than they otherwise would — is potentially more important, but as discussed in Ap- 
pendix C, seems unlikely to greatly weaken the positive findings appearing in Table 3.1. Cer- 
tainly, they are very unlikely to become negative. 


St. Nicks Alliance 


The benefit-cost findings for St. Nicks Alliance appear in Table 3.3. This table is very 
similar to Table 3.1 for Per Scholas — program effects on time available outside of work and on 
deadweight loss are omitted and the table pertains only to the observation period. Although par- 
ticipants appear to have enjoyed modest net gains during the five-and-three-quarter-year obser- 
vation period, the cost of operating WorkAdvance at St. Nicks Alliance was larger than increases 
in tax payments and reductions in transfer payments, and as a result, the government’s budgetary 
position diminished, and society was worse off. These bottom-line figures look even worse when 
deadweight loss, which increased because the government’s budgetary position was worse, and 
the loss of time outside of work are considered (see Table 3.2). 


The picture changes considerably when the benefits and costs are projected beyond the 
observation period. Looking at a 10-year period (that is, a projection of four and a quarter years 
beyond the observation period) and including deadweight loss and the value of time outside of 
work, the net gain of participants increases from $1,623 to $9,387, the previously negative change 
in the government’s budgetary position becomes positive at $3,660, and the loss to society be- 
comes a gain of $13,742 (see Table 3.2). These rather dramatic results occur because, unlike the 
other three sites, earnings impacts at St. Nicks Alliance were still increasing at the end of the 
observation period, albeit modestly, and this is reflected in the projections. These projections are 
of course, subject to error. That is why the projection period was limited to a little over four years. 
However, even if the projections are limited to only a year and a half beyond the observation 
period, and hence are presumably less subject to error, net benefits at the St. Nicks Alliance site 
are still positive from the societal perspective. As in the case of Per Scholas, the analysis in Ap- 
pendix C suggests that these positive results for St. Nicks Alliance are unlikely to diminish sub- 
stantially if the benefits and costs that are not estimated in dollars were somehow incorporated 
into the net gain estimates. 
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Table 3.3 


Benefits and Costs for St. Nicks Alliance by Accounting Perspective 
Over the 63-Month Observation Period (in 2018 dollars) 


Government 

Benefits and costs ($) Participants Budget Society 
Pretax earnings of participants 5,197 0 5,197 
Fringe benefits 1,298 0 1,298 
Payroll taxes -853 853 0 
Income taxes -894 894 0 
Sales taxes -99 99 0 
Work-related expenditures -392 0 -392 
TANF 545 -545 0 
SNAP -634 634 0 
SSI/SSDI -1,307 1,307 0 
Housing assistance -1,012 1,012 0 
Ul 950 -950 0 
Administrative cost of transfer programs 0 93 93 
Net program costs 0 -7,527 -7,527 
Net financial gains or losses without 

nonmarket time and deadweight loss 2,799 -4,130 -1,331 
Nonmarket time -1,176 0 -1,176 
Deadweight loss 0 0 -786 
Net financial gains or losses with 

nonmarket time and deadweight loss 1,623 -4,130 -3,293 


SOURCES: The sources and derivation of net program costs are described in Hendra et al. 
(2016). The sources and derivation of the remaining benefit and cost components are described 
in Appendix B. 


NOTES: All benefits and costs are in dollars per participant over five years and are inflation- 
adjusted to 2018 dollars and discounted to 2018 present values. 

TANF = Temporary Assistance for Needy Families, SNAP = Supplemental Nutrition 
Assistance Program, SSI = Supplemental Security Income, SSDI = Social Security Disability 
Insurance, UI = Unemployment Insurance 

Rounding may cause slight discrepancies in sums and differences. 


Madison Strategies Group 


The benefit-cost findings for the Madison Strategies Group are shown in Table 3.4. These 
findings are largely driven by the large impact of nearly $20,000 on pretax earnings during the 
observation period. However, as discussed in Appendix B, it appeared that the earnings impact 
did not extend beyond the site’s five-and-a-half-year observation period. Thus, projections be- 
yond this period were unnecessary. 
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Table 3.4 


Benefits and Costs for Madison Strategies Group by Accounting 
Perspective Over the 62-Month Observation Period (in 2018 dollars) 


Benefits and costs ($) 

Pretax earnings of participants 
Fringe benefits 

Payroll taxes 

Income taxes 

Sales taxes 

Work-related expenditures 
TANF 

SNAP 

SSI/SSDI 

Housing assistance 

Ul 

Administrative cost of transfer programs 


Net program costs 


Net financial gains or losses without 


nonmarket time and deadweight loss 


Nonmarket time 


Deadweight loss 


Net financial gains or losses with 


nonmarket time and deadweight loss 


Participants 
19,808 
4,948 
-3,253 
-3,137 
-824 
-1,495 
13 
-468 
-523 
773 
-168 


11,192 


Government 
Budget 


1,615 


Society 
19,808 
4,948 


13,114 


SOURCES: The sources and derivation of net program costs are described in Hendra et al. 
(2016). The sources and derivation of the remaining benefit and cost components are described 


in Appendix B. 


NOTES: All benefits and costs are in dollars per participant over five years and are inflation- 
adjusted to 2018 dollars and discounted to 2018 present values. 


TANF = Temporary Assistance for Needy Families, SNAP = Supplemental Nutrition 


Assistance Program, SSI = Supplemental Security Income, SSDI = Social Security Disability 


Insurance, Ul = Unemployment Insurance 


Rounding may cause slight discrepancies in sums and differences. 


Because of the large impact on pretax earnings, the fringe benefits received by partici- 
pants increased. However, their work-related expenditures and tax payments also increased, but 
not by nearly enough to offset their increases in earnings and fringe benefits. WorkAdvance’s 
effects on transfer payments appear to be very modest at Madison Strategies Group. Thus, pro- 
gram participants enjoyed substantial net gains, as did society as a whole. The increases in tax 
payments were sufficient to offset the cost of operating WorkAdvance at Madison Strategies 
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Group, resulting in a modest improvement in the government’s budgetary position. Because the 
program’s effect on the government’s budgetary position was modest, deadweight loss was esti- 
mated to decline by only $307. However, there appears to have been a large loss in time outside 
of work, which decreased the net gain of participants from $15,673 to $11,192. Once again, it 
appears that if the value of the benefits and costs that could not be measured in dollars could be 
incorporated into the net gain estimates, the positive findings for Madison Strategies Group would 
be unlikely to become negative. Because the unemployment rate remained low throughout the 
observation period in Tulsa, where Madison Strategies Group is located, any losses of earnings 
by nonparticipants would be expected to be especially small. 


Towards Employment 


Table 3.5 presents the benefit-cost findings for Towards Employment. Like Madison 
Strategies Group, Towards Employment did not seem to have effects on pretax earnings that ex- 
tended beyond the five-and-a-half-year observation period at that site. Thus, projections were not 
made for WorkAdvance at the Towards Employment site. Moreover, the impact on pretax earn- 
ings at the site was smaller than at Madison Strategies Group or Per Scholas. As a result, the 
bottom-line net gains for participants and society that appear in Table 3.5 are also smaller, alt- 
hough still positive and substantial. However, the increase in pretax earnings only generated suf- 
ficient increases in tax revenues and decreases in transfer payments to roughly offset WorkAd- 
vance’s operating costs of nearly $6,000 per participant. Thus, the government’s budget was 
essentially unaffected.° Given this, the program’s effect on deadweight loss at Towards Employ- 
ment was negligible, but the estimate of the value of the loss of time outside the workplace is 
substantial. As is the case for the other three sites, the analysis in Appendix C suggests that if it 
were possible to include the nonmonetized benefits and costs, it is unlikely that the positive net 
gains shown in Table 3.5 would diminish by a substantial amount, and it is even more unlikely 
that they would become negative. 


Overall 


The bottom-line net gain findings for the four sites are summarized in Table 3.2, which 
includes the cost of lost nonwork time and deadweight loss. The table reports the bottom-line 


®As described in Appendix B, the assumptions needed to estimate the program’s impact on government 
transfer benefits are especially tenuous. As a result, these estimates are potentially subject to error, possibly caus- 
ing the estimated net financial gains of the government to be substantially overstated. To illustrate the implication 
of this, if the impact on government transfer benefits was zero, net financial benefits from the government’s 
perspective at the Towards Employment site would become negative and those at the St. Nicks Alliance site 
would fall to near zero. Net financial gains from the participant perspective, however, would increase by the 
corresponding amount and, consequently, net financial gains from the societal perspective would be unchanged. 
Of course, given the positive impacts of WorkAdvance on employment and earnings at all four sites, it seems 
likely that the program did cause some reduction in the use of government transfer benefits, although possibly 
by less than the estimated amounts. 
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Benefits and Costs for Towards Employment by Accounting 


Table 3.5 


Perspective Over the 62-Month Observation Period (in 2018 dollars) 


Benefits and costs ($) 

Pretax earnings of participants 
Fringe benefits 

Payroll taxes 

Income taxes 

Sales taxes 

Work-related expenditures 
TANF 

SNAP 

SSI/SSDI 

Housing assistance 

Ul 

Administrative cost of transfer programs 


Net program costs 


Net financial gains or losses without 


nonmarket time and deadweight loss 


Nonmarket time 


Deadweight loss 


Net financial gains or losses with 
nonmarket time and deadweight loss 


Participants 
12,638 
3,157 
-2,075 
-1,773 
-398 
-954 
-872 
-1,028 
-886 
1,370 
-242 


5,505 


Government 
Budget 


265 


265 


Society 
12,638 
3,157 


5,820 


SOURCES: The sources and derivation of net program costs are described in Hendra et al. 
(2016). The sources and derivation of the remaining benefit and cost components are described 


in Appendix B. 


NOTES: All benefits and costs are in dollars per participant over five years and are inflation- 
adjusted to 2018 dollars and discounted to 2018 present values. 


TANF = Temporary Assistance for Needy Families, SNAP = Supplemental Nutrition 


Assistance Program, SSI = Supplemental Security Income, SSDI = Social Security Disability 


Insurance, UI = Unemployment Insurance 
Rounding may cause slight discrepancies in sums and differences. 


Focusing on the findings that combine the observation and projection periods, the results 
indicate that because of increases in earnings and fringe benefits, WorkAdvance participants, on 
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findings for the observation period alone and for the combined observation and projection peri- 
ods. As previously indicated, the findings for the combined periods are based on benefits and 
costs that accrued over five and a half years for Towards Employment and Madison Strategies 
Group, seven years for Per Scholas, and ten years for St Nicks Alliance. 


average, made very substantial gains of more than $5,000 to more than $15,000 even though they 
paid higher taxes and, at some sites, relinquished appreciable amounts of government transfer 
benefits. As Tables 3.1 and 3.3-3.5 indicate, the net gains are even greater if the cost of lost non- 
work time is ignored. Although considerable costs were incurred in operating WorkAdvance, 
these costs were offset at one site, and more than offset at the other three sites, by increases in tax 
payments received from participants and reductions in transfer benefits paid to participants. How- 
ever, the net gains for the government’s budget at Madison Strategies Group were small and those 
at Towards Employment were negligible.” For every dollar the government invested in program 
operating costs, its return was $4 at Per Scholas, $1.49 at St. Nicks Alliance, $1.27 at Madison 
Strategies Group, and $1.04 at Towards Employment. Because participants were better off at all 
four sites and the government’s budget also improved at three sites, the gains for society as a 
whole from all four programs were substantial: nearly $6,000 at Towards Employment; over 
$13,000 at St. Nicks Alliance and Madison Strategies Group; and $31,000 at Per Scholas if the 
costs of losses in nonmarket time and the change in deadweight loss are included, and even larger 
if these items are ignored. These findings and the estimates of program operating costs imply that 
for every dollar of costs needed to operate WorkAdvance, society reaped eight dollars at the Per 
Scholas site, around three dollars at the St. Nicks Alliance and Madison Strategies Group sites, 
and two dollars at Towards Employment. It appears likely that the findings summarized in this 
paragraph would still hold up even if the estimates of net gains included those benefits and costs 
that could not be measured in dollars. 


These net gain estimates are exceptional. Benefit-cost findings for employment and train- 
ing programs for the disadvantaged seldom find net gains of more than a few thousand dollars, 
and finding net losses is not infrequent.® 


Accounting for Sampling Variation Through Monte Carlo Analysis 


Just as the impact estimates presented earlier in this report are subject to uncertainty due to sam- 
pling variation (see Box 2.2), the estimates of the benefits and costs of WorkAdvance are also 
subject to uncertainty resulting from sampling variation. In other words, if different samples of 
individuals had been assigned to the WorkAdvance and control groups, the impact estimates upon 
which the benefits and costs are based may have differed. This shows up in the extent to which 


TAs previously indicated, program operating costs are treated in the benefit-cost analysis as ifthey were paid 
for by the government, although a large share of operating costs was actually paid for by foundations. 

8For example, a random assignment benefit-cost study of the Job Training Partnership Act found net gains 
of $3,024 for adult men and $3,875 for adult women, but net losses from the government perspective of $1,935 
and $2,760, respectively, resulting in net gains for society of $1,084 and $1,114, respectively. The findings for 
youth were negative from all three perspectives (Orr et al. 1996). A random assignment benefit-cost study of the 
Job Corps found net losses for society of $12,755 when estimated for all program participants, but net gains for 
society of $20,870 when the analysis was limited to participants ages 20 to 24 (Schochet, Burghardt, and 
McConnell (2006). A meta-analysis of random-assignment benefit-cost studies of welfare-to-work programs 
with 50 observations found that the median gain from the perspectives of participants, the government, and so- 
ciety, were $283, $123, and $640, respectively (Greenberg and Cebulla, 2008). All the values in this footnote 
have been converted to 2018 prices using the Consumer Price Index. 
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the impacts estimates are statistically significant. To address this uncertainty, MDRC conducted 
a Monte Carlo analysis of the impact estimates upon which the benefit-cost analysis depends. As 
discussed earlier and in Appendix B, the impact estimates of earnings are an especially crucial 
ingredient in the benefit-cost estimates. The Monte Carlo analysis recognizes that the earnings 
impact point estimate is the best available indicator of the program’s true impact, but it is not the 
only estimate. The true impact could be larger or smaller than the point estimate. If those alterna- 
tive estimates typically show a positive impact, this would add some assurance that the true 1m- 
pact of the program is indeed positive (even if the point estimate impact is not statistically signif- 
icant). For the WorkAdvance study, the Monte Carlo analysis involved randomly drawing 2,000 
impact estimates from among all possible estimates within the normal distribution implied by the 
standard errors of the impact estimates.? 


To understand the procedure followed in the Monte Carlo analysis, consider how the 
2,000 random draws are done for one of the estimates of the positive impact of WorkAdvance on 
earnings. The standard error of the impact estimate implies that each random draw would vary 
along a normal (bell-shaped) curve. In a graph of that curve, the X-axis would represent all the 
possible dollar amounts resulting from the 2,000 random draws and the Y-axis would represent 
the probability that a particular dollar amount would result from a given random draw. The sum 
of the probabilities would be 100 percent. The highest point on the curve would be the original 
point estimate of the earnings impact. Thus, it would have the highest probability of being ran- 
domly drawn. Given the bell shape of the curve, as one moves away from the highest point, the 
probabilities shrink. Thus, near the ends of the curve, the probabilities are smaller than they are 
in the middle of the curve. They can still be drawn, but they will be drawn less often than dollar 
values nearer the highest point. 


As previously indicated, the Monte Carlo analysis randomly draws 2,000 values of the 
eamings impacts. The values drawn depend on the shape and location of the curve, and this, in 
turn, is determined by the size of the original impact estimate and its standard error. Because the 
normal curve is symmetrical on each side of its highest point, the average of the 2,000 random 
draws should be very close to the value of the original impact estimate. Because of the bell shape 
of the normal curve, there will be more draws near the original point estimate than near the tails 
of the curve. The larger the standard errors relative to the original impact estimate (that is, the 
lower the level of statistical significance), the more likely it is that the normal curve overlaps zero 
and the more draws there will be that are below zero, implying a higher probability that the true 


°This was not done for the estimates of WorkAdvance’s operating costs because their standard errors are 
unknown. Instead, identical estimates of operating costs are used in each of the 2,000 draws. It was done for two 
other sets of impact estimates used in the analysis: program impacts on earnings and an impact measure that was 
computed by summing the estimated individual program impacts on various government transfer benefits. Once 
an earnings impact and a transfer payment impact were randomly drawn, these values were used to compute the 
remaining benefits and costs (for example, fringe benefits, tax payments, and the administrative costs of govern- 
ment transfers) by using the methods described in Appendix B. It is possible that the impacts on earnings and on 
transfer payments are correlated. Because they were estimated from different data sets, however, it was not pos- 
sible to account for this in the Monte Carlo analysis. Thus, the two sets of impacts are necessarily treated as if 
they are independent. At least one study author found that his Monte Carlo findings were insensitive to this 
assumption (Jerome, 2012). 
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impact is not positive. Seen from the opposite point of view, the greater the proportion of possible 
impact estimates (that is, the greater the proportion of the 2,000 draws) that are positive, the more 
confidence one can have that the true impact is positive. 


Each draw of the various benefit and cost impacts provides the information needed to 
conduct a benefit-cost analysis. In essence, with 2,000 draws, 2,000 separate benefit-cost analyses 
can be conducted. In performing the Monte Carlo analysis, once the random draws were made, 
total net gains (or losses) from the participant, government, and social perspectives were com- 
puted 2,000 times, once for each set of draws. The means of the resulting 2,000 estimates of net 
gains (or losses) and their standard deviations were then computed. The size of the standard de- 
viations relative to their means indicates the uncertainty pertaining to the means. The proportion 
of the net gain (or loss) estimates that are positive indicate the probability that the WorkAdvance 
program in a given site was cost-beneficial (that is, the probability that a net gain rather than a net 
loss occurred) and the proportion that are negative is a measure of the probability that it was not. 


Results from the Monte Carlo analyses of each site from each of the three perspectives 
appear in Table 3.6. The top panel pertains to all the estimated benefits and costs, including those 
projected beyond the observation period, while the bottom panel is limited to only benefits and 
costs estimated for the observation period. The findings in the two panels for the Madison Strat- 
egies Group and Towards Employment sites are identical because no projections were made for 
these sites. The top row in each panel shows the base case estimates of net gains (including the 
estimates of deadweight loss and the value of lost nonwork time) for each site that were reported 
above, while the remaining rows were obtained from the Monte Carlo analysis. As expected, the 
table indicates that the original estimates of net gains and those derived by averaging the net gains 
over the 2,000 estimates from random draws are very similar. Of greater importance, except for 
Per Scholas from all three perspectives and Madison Strategies Group from the participant per- 
spective, the standard deviations of the estimates of the average net gains are fairly large relative 
to the averages themselves. Consequently, other than these four average net gains, the averages 
are not statistically significant at the 95 percent confidence level, implying that the true net gains 
could be considerably larger or smaller. Indeed, it is even possible that they are negative (that 1s, 
that net losses occurred). For the most part, this imprecision in or uncertainty about the averages 
is due to the rather small samples on which they are based. 


Even in the face of these relatively large standard deviations, however, the top panel of 
Table 3.6 implies that the probability that there were positive net gains is typically considerably 
larger than the probability that there were net losses. For example, at all four sites, the probability 
of positive net gains was over 80 percent from the participant perspective and at least 75 percent 
from the perspective of society as a whole. These probabilities were nearly 100 percent at the Per 
Scholas site. As the bottom panel of Table 3.6 suggests, except for the St. Nicks Alliance site, 
these results also hold up when based on only benefits and costs that are estimated for the obser- 
vation period alone. Thus, when viewed from the participant and societal perspectives, it is quite 
likely (although not certain) that WorkAdvance produced net gains at three of the four sites. Ig- 
noring possible benefits at St. Nicks Alliance after the end of the observation period, Table 3.6 
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Table 3.6 


Summary Statistics from the Monte Carlo Analysis of WorkAdvance by Accounting Perspective 
Net Financial Gains or Losses (in 2018 Dollars), by Site 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 
Government Government Government Government 
Participants budget Society Participants budget Society Participants budget Society Participants budget Society 


Results that include projections 


beyond the observation period 
Base case ($) 15,456 13,387 31,387 9,387 3,660 13,742 11,192 1,615 13,114 5,505 265 5,820 


Average net gains from 2,000 
sampling draws ($) 15,120 13,311 30,959 9,449 3,648 13,790 11,180 1,711 13,216 5,321 424 5,825 


Standard deviation of net 


gains from 2,000 


sampling draws ($) 7,159 4,871 11,692 7,190 5,178 12,112 6,134 4,107 10,602 6,040 4,926 8,708 
Probability of net gains (%) 98.1 99.6 99.6 91.0 76.7 87.1 96.8 66.2 89.3 81.6 53.2 75.1 
Probability of net losses (%) 1.9 0.5 0.5 9.1 23.4 13.0 3.3 33.8 10.7 18.5 46.8 24.9 


Results for the observation period 
Base case ($) 13,997 11,370 27,535 1,623 -4,130 -3,293 11,192 1,615 13,114 5,505 265 5,820 


Average net gains from 2,000 
sampling draws ($) 13,997 11,405 27,569 1,708 -4,115 -3,188 11,180 1,711 13,216 5,321 424 5,825 


Standard deviation of net 
gains from 2,000 
sampling draws ($) 6,381 4,363 10,264 1,472 98 1,588 6,134 4,107 10,602 6,040 4,926 8,708 


(continued) 
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Table 3.6 (continued) 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 
Government Government Government Government 


Participants budget Society Participants budget Society Participants budget Society Participants budget Society 


Probability of net gains (%) 98.3 99.3 99.5 60.5 18.7 37.2 96.6 65.6 89.1 82.8 50.8 74.4 
Probability of net losses (%) 1.8 0.8 0.6 39.5 81.3 62.8 3.4 34.4 10.9 17.3 49.3 25.6 


SOURCES: The sources and derivation of net program costs are described in Hendra et al. (2016). The sources and derivation of the remaining benefit and cost 
components are described in Appendix B. 


NOTES: All benefits and costs are in dollars per participant over five years and are inflation-adjusted to 2018 dollars and discounted to 2018 present values. 
The length of the observation period for the Per Scholas and St. Nicks Alliance sites is 63 months and the observation period for the Madison Strategies 
Group and the Towards Employment sites is 62 months. 
Rounding may cause slight discrepancies in sums and differences. 


implies that there is nearly a 40 percent probability that there were net losses for participants at the site and more than a 60 percent probability that 
there were net losses from the societal perspective. However, since it seems likely that at least some benefits and costs for St. Nicks Alliance did 
occur after the end of the observation period, the probabilities of net losses are probably lower than 40 and 60 percent, respectively, and perhaps 


much lower. 


Although WorkAdvance’s impact on the government’s fiscal position was almost surely positive at Per Scholas, it is difficult to draw firm 
conclusions from the Monte Carlo analysis about whether there were net gains or losses from the government’s perspective in the other three sites. 
For example, unless the government reaped fairly large gains after the observation period ended at St. Nicks Alliance, which is far from certain, the 
government’s budgetary position may have worsened as a result of running WorkAdvance at that site. There is almost a 35 percent probability that 
there were net losses in the government’s budget for the program at the Madison Strategies Group site and the probability that there were net losses 
in the government’s budgetary position for the program at the Towards Employment site is almost as high as the probability that there were positive 
net gains. Still, it is unlikely that the government’s budget suffered large losses relative to the gains for participants at these two sites. Only about an 
eighth of the draws for Madison Strategies Group and a quarter of the draws for Towards Employment imply net losses in the government’s budget 
of more than $3,000. In contrast, over nine-tenths of the draws for Madison Strategies Group and two-thirds of the draws for Towards Employment 
imply that the net gains for program participants in these sites were greater than $3,000. 


Chapter 4 


Conclusion 


The findings presented in this report contribute to the growing body of evidence on the effective- 
ness of sector programs. Specifically, these findings offer some of the first rigorous evidence of 
how effective these programs are at promoting upward mobility in the long term (between Years 
4 and 8 after individuals enter programs) and of the benefits and costs of operating such programs. 


WorkAdvance was an attempt to initially increase employment and eventually to increase 
earnings and help participants advance along a career pathway. This report shows that the previ- 
ous increases in employment seen at some of the sites faded in the long term; none of the sites 
increased employment by a statistically significant amount in either 2017 or 2018. However, the 
long-term impact findings show evidence of earnings increases at some sites and for some sam- 
ples. Per Scholas produced large impacts on average earnings in both 2017 and 2018 and an im- 
pact on the likelihood of individuals having high earnings in 2018. St. Nicks Alliance did not 
produce statistically significant impacts on average earnings in either year. However, the site in- 
creased the likelihood of individuals having high earnings in 2018. There were no statistically 
significant impacts on average earnings among the late cohorts or full samples at Madison Strat- 
egies Group and Towards Employment in either year. However, both sites increased the likeli- 
hood of individuals having high earnings: at Madison Strategies Group, among the late cohort in 
2017 and among the full sample in both years; and at Towards Employment, among the late 
cohort in 2017. The pattern of findings suggests that the earnings-based impacts are driven by 
WorkAdvance group members having higher wages than control group members (rather than by 
being employed at a higher rate),! and that therefore, they likely advanced in their careers over 
time. 


Among the pooled sample, WorkAdvance increased earnings by statistically significant 
amounts in both years. These impacts are driven in part by the earnings impacts at Per Scholas, 
but the impacts do not seem to be limited to only that site (there were earnings impacts in some 
quarters and years among the pooled sample from the other three sites). Additionally, WorkAd- 
vance was successful in helping the semiattached and long-term unemployed groups reenter the 
labor market and increase their earnings. The earnings impacts among the semiattached group are 
quite large. 


The findings from the benefit-cost analysis are positive from the perspectives of Work- 
Advance participants, the government, and society at all four sites. Thanks to increases in earnings 
and fringe benefits, WorkAdvance group members made very substantial financial gains of 


'The increases in earnings could also be driven by increases in hours worked. The National Directory of 
New Hires data do not include information on hours worked, so it is not possible to test how much of the earnings 
impacts, if any, are attributable to hours worked. However, an analysis based on the Year 2 survey data showed 
that around half or more of WorkAdvance’s impact on earnings at each site was attributable to hourly wages, 
with the rest attributable to hours worked. 
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between $5,500 and $15,500 over the observation and projection period (which ranges from 5 
years to 10 years), even though they paid higher taxes and gave up appreciable amounts of gov- 
ernment transfer benefits. And although the government incurred considerable costs in operating 
WorkAdvance, these costs were at least offset at all four sites by participants paying more in taxes 
and receiving less in government transfer benefits.* Because participants were better off at all four 
sites and the government’s budget also improved, the financial gain for society as a whole from 
all four programs was substantial. These positive benefit-cost findings are not often seen in eval- 
uations of employment and training programs. 


Overall, the WorkAdvance results support the case for focusing on how sector programs 
can be improved. The long-term economic impacts of WorkAdvance — as well as findings from 
other evaluations of sector programs* — show that sector programs can increase earnings in the 
longer term and can lead to advancement gains over time for low-income individuals. At the same 
time, sector programs can be hard to implement well and not all programs will lead to statistically 
significant increases in employment and earnings. Therefore, it seems prudent to focus future 
efforts on how to make the sectoral approach more consistently successful so that workforce pro- 
viders can implement new sector-based programs or continue to strengthen their current ones. In 
particular, future effects should reflect and gather evidence on how programs can continue to 
refine their advancement-focused services, including those offered to participants after they have 
obtained employment, in order to better support participants as they move up career pathways. 


This is the final planned report for the WorkAdvance evaluation. Longer-term findings 
from several other evaluations of sector programs will be released in the next few years, which 
will provide more evidence on the effectiveness of sector strategies. 


As discussed in Chapter 3, results from the Monte Carlo analysis suggest that it is difficult to know for 
certain whether there were net losses or net gains from the government’s perspective at St. Nicks Alliance, Mad- 
ison Strategies Group, and Towards Employment, although whatever the direction, they were probably small. 


3Roder and Elliott (2019). 
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Appendix A 


Additional Economic Impact Analyses 


This appendix presents findings from the original WorkAdvance study’s power analysis, offers 
additional findings from the economic impact analysis based on state unemployment insurance 
wage data and National Directory of New Hires (NDNH) data, and describes the similarities and 
differences between the two sources of employment and earnings data. 


Power Analysis 


Appendix Table A.1 shows the minimum detectable effects (MDEs) calculated during the power 
analysis done for the original WorkAdvance evaluation.! MDEs are a key measure of statistical 
power. Conventionally, an MDE is the smallest true effect that has an 80 percent chance of being 
statistically significant at the 10 percent level. MDEs are commonly expressed in effect size units 
(specifically, in terms of standard deviations) to permit comparisons across outcomes with differ- 
ent units. This expression of an MDE is referred to as a minimum detectable effect size (MDES). 
A common rule of thumb is to ensure studies have sufficient power to detect impacts at or below 
an MDES of 0.2, which is a common threshold for a “small” effect size.? 


As shown in Appendix Table A.1, the MDESs at the four WorkAdvance providers are 
between 0.167 (at Towards Employment) and 0.204 (at St. Nicks Alliance). These are all below 
or close to the 0.2 threshold. Assuming 50 percent of the control group was employed (that is, the 
standard deviation is 0.5),* these MDESs translate into MDEs of between 8.4 and 10.2 for per- 
centage measures. 


Differences Between State Unemployment Insurance Data and 
NDNH Data 


There are a few differences between state unemployment insurance wage data and 
NDNH data in terms of coverage.* For example, state unemployment insurance wage data only 
contain employment and earnings records for employers in that state, while NDNH data contain 
employment and earnings records for employers in all 50 states. In the case of WorkAdvance, 
this may lead to differences in reporting for the providers that are located close to state borders, 
such as Per Scholas and St. Nicks Alliance (both are located in New York but are close to New 
Jersey and Connecticut), if many sample members are working in a state other than the state in 
which the provider is located. Additionally, while both data sources include workers who are 


'Hendra et al. (2016). 

The 0.2 rule of thumb comes from Cohen. Cohen defined an effect size of 0.2 as “small,” 0.5 as “medium,” 
and 0.8 as “large.” Lipsey, another prominent researcher, sets the threshold lower. To Lipsey, an effect size of 
0.15 or lower is small. See Cohen (1992) and Hill, Bloom, Black, and Lipsey (2007). 

>This assumption is the worst-case scenario. The point of maximum variance for a percentage measure is 
0.5 (a control group level of 50 percent). At that point, an MDES of 0.2 translates into an MDE of 10 percent- 
age points. The further the variance is from 0.5, the smaller the MDE. For example, if the control group level for 
a measure is 20 percent, the MDE for a study powered at 80 percent would be 8 percentage points. 

4See Czajka, Patnaik, and Negoita (2018) for more information on using state unemployment insurance 
wage and NDNH data as sources of earnings data in research evaluations. 
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covered by unemployment insurance, the NDNH data also include federal workers and, from 
some states, workers who are self-employed. 


State Unemployment Insurance Data Impacts, by Site 


Previous WorkAdvance reports presented employment and earnings impacts based on 
state unemployment insurance wage and benefits data through Year 3. For this report, the evalu- 
ation team was able to collect additional state unemployment insurance wage data from New 
York (covering sample members from Per Scholas and St. Nicks Alliance) and Ohio (covering 
sample members from Towards Employment) through Year 5. However, the evaluation team was 
not able to collect additional state unemployment insurance wage data from Oklahoma, so data 
are only available through Year 3 for sample members from Madison Strategies Group. Appendix 
Tables A.2 through A.5 present impacts for each site based on the state unemployment insurance 
wage data. 


State Unemployment Insurance Data Impacts on Employment 
and Earnings in the Targeted Sector 


The state unemployment insurance wage data from Ohio and Oklahoma included North 
American Industry Classification System (NAICS) codes that are linked to employers and define 
the sector in which the employer operates. These codes were not available in the state unemploy- 
ment insurance wage data from New York or in the NDNH data. Appendix Figure A.1 shows 
impacts on employment in the targeted sector by site and random assignment sector. 


The top graph in Appendix Figure A.1 shows impacts on unemployment insurance-cov- 
ered employment in the targeted sectors — transportation and manufacturing — at Madison Strat- 
egies Group. Beginning in the quarter of random assignment, employment in the targeted sectors 
increased for both research groups, but WorkAdvance group members were significantly more 
likely than control group members to be employed in the targeted sectors through Quarter 12. 


The bottom two graphs in Appendix Figure A.1 show that at Towards Employment, the 
impacts on working in the manufacturing sector (among those who initially targeted the manu- 
facturing sector) are larger and more consistent than the impacts on working in the health care 
sector (among those who initially targeted the health care sector). WorkAdvance increased em- 
ployment in the manufacturing sector (among sample members randomly assigned into the man- 
ufacturing sector program) from Quarter 3 to Quarter 10, from Quarter 13 to Quarter 16, and from 
Quarter 19 to Quarter 21. The percentage of Towards Employment sample members working in 
the health care sector (among those randomly assigned into the health care sector program) was 
fairly consistent across research groups throughout most of the follow-up period. The fact that as 
many control group members as WorkAdvance group members found jobs in health care suggests 
that it was easier for individuals who were not eligible to receive WorkAdvance services to obtain 
jobs in the health care sector — either on their own or by getting training and support somewhere 
else in the community — than to obtain jobs in the manufacturing sector. 
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Appendix Table A.1 


Minimum Detectable Effects, by Site 


Sample 
Sample Size 
Per Scholas 690 
St. Nicks Alliance 479 
Madison Strategies Group 697 
Towards Employment 698 


R-squared* 


0.183 
0.193 
0.142 
0.219 


SOURCE: MDRC calculations using PowerUP! tool. 


NOTES: MDE = minimum detectable effect; MDES = minimum detectable effect size; SD = standard deviation. 


MDES 


0.171 
0.204 
0.175 
0.167 


Employment (%) 


SD=0.4 SD=0.5 


6.8 
8.2 
7.0 
6.7 


8.6 
10.2 
8.8 
8.4 


MDE? 


Annual earnings ($) 
SD=8,000 SD = 14,000 


1,368 
1,632 
1,400 
1,336 


aR-squared values are from the models for unemployment insurance earnings in Quarter 10. 
>MDEs are for a two-tailed test at the 10 percent significance level with 80 percent power. 
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2,394 
2,856 
2,450 
2,338 


Appendix Table A.2 


Per Scholas Impacts on Employment and Earnings 
in Years 1 to 5 


WorkAdvance Control Difference 


Group Group (Impact) P-Value 


Among the full sample 


Year 1 earnings ($) 8,868 8,718 150 0.846 
Year 2 earnings ($) 18,218 14,474 3,744 **™* 0.002 
Year 3 earnings ($) 23,095 17,711 5,384 *** 0.000 
Ever employed in Year 4 (%) 76.4 72.8 3.6 0.280 
Year 4 earnings ($) 25,527 20,624 4,903 *** 0.005 
Ever employed in Year 5 (%) 73.3 73.4 -0.1 0.980 
Year 5 earnings ($) 27,897 24,439 3,458 * 0.078 


Total earnings 
Years 1 to 5 ($) 103,604 85,966 17,639 *** 0.003 


Full site sample size 349 341 


Among the early cohort? 
Year 4 earnings ($) 25,145 18,368 6,777 *** 0.004 
Year 5 earnings ($) 28,926 22,153 6,773 *** 0.007 


Among the late cohort? 


Year 4 earnings ($) 25,458 23,835 1,623 0.556 
Year 5 earnings ($) 26,339 27,501 -1,162 0.714 
Early cohort sample size 189 185 
Late cohort sample size 160 156 


SOURCE: MDRC calculations from unemployment insurance 
administrative records provided by the New York State Department of 
Labor. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

aThe difference between cohort subgroups for Year 4 earnings is not 
statistically significant, and for Year 5 is statistically significant at the 10 
percent level (indicated by gray shading). 

Rounding may cause slight discrepancies in sums and differences 
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Appendix Table A.3 


St. Nicks Alliance Impacts on Employment 


and Earnings in Years 1 to 5 


WorkAdvance Control 


Among the full sample 


Year 1 earnings ($) 
Year 2 earnings ($) 
Year 3 earnings ($) 


Ever employed in Year 4 (%) 


Year 4 earnings ($) 


Ever employed in Year 5 (%) 


Year 5 earnings ($) 


Total earnings 
Years 1 to 5 ($) 


Full site sample size 


Among the early cohort? 
Year 4 earnings ($) 


Year 5 earnings ($) 


Among the late cohort? 
Year 4 earnings ($) 


Year 5 earnings ($) 


Early cohort sample size 


Late cohort sample size 


Group 


9,395 
14,420 
16,787 


69.8 
19,393 


70.1 
20,960 


80,956 


242 


17,037 
19,140 


22,059 
22,879 


127 
115 


Group 


9,648 
14,229 
17,260 


70.5 
18,175 


70.3 
19,078 


78,390 


237 


16,918 
19,291 


19,660 
18,913 


131 
106 


Difference 


(Impact) P-Value 


-253 
191 
-473 


-0.7 
1,217 


-0.2 


1,882 


2,566 


120 


-150 


2,399 
3,966 


0.802 


0.892 


0.771 


0.871 
0.517 


0.961 
0.363 


0.704 


0.961 


0.956 


0.419 
0.220 


SOURCE: MDRC calculations from unemployment insurance administrative 


records provided by the New York State Department of Labor. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 


percent; ** = 5 percent; * = 10 percent. 
@The difference between cohort subgroups is not statistically significant 


for either measure. 


Rounding may cause slight discrepancies in sums and differences. 
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Appendix Table A.4 


Madison Strategies Group Impacts on Employment 
and Earnings in Years 1 to 3 


WorkAdvance_ Control 


Group Group Impact P-Value 


Among the full sample 


Year 1 earnings ($) 13,261 12,933 328 0.682 
Year 2 earnings ($) 16,640 14,822 1,818 * 0.085 
Year 3 earnings ($) 16,197 14,826 1,371 0.225 
Ever employed in Year 4 (%) N/A N/A N/A 


Year 4 earnings ($) 


Ever employed in Year 5 (%) N/A N/A N/A 
Year 5 earnings ($) N/A N/AA NIA 


Total earnings 
Years 1 to 3 ($) 46,098 42,581 3,517 0.177 


Full site sample size 353 344 


Among the early cohort 
Year 4 earnings ($) N/A N/A N/A 
Year 5 earnings ($) N/A N/AA N/A 


Among the late cohort 


Year 4 earnings ($) N/A N/AA N/A 
Year 5 earnings ($) N/A N/A N/A 
Early cohort sample size 173 164 
Late cohort sample size 180 180 


SOURCE: MDRC calculations from unemployment insurance administrative 
records provided by the Oklahoma Employment Security Commission. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

State-based unemployment insurance administrative records are not 
available for Years 4 and 5. 

Rounding may cause slight discrepancies in sums and differences. 
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Appendix Table A.5 


Towards Employment Impacts on Employment 


and Earnings in Years 1 to 5 


Among the full sample 


Year 1 earnings ($) 
Year 2 earnings ($) 
Year 3 earnings ($) 


Ever employed in Year 4 (%) 
Year 4 earnings ($) 


Ever employed in Year 5 (%) 
Year 5 earnings ($) 


Total earnings 
Years 1 to 5 ($) 


Full site sample size 


Among the early cohort* 
Year 4 earnings ($) 
Year 5 earnings ($) 


Among the late cohort? 
Year 4 earnings ($) 
Year 5 earnings ($) 


Early cohort sample size 


Late cohort sample size 


WorkAdvance Control 


Group 


9,495 
13,230 
14,202 


79.2 
16,173 


78.2 
17,213 


70,314 


349 


12,671 
14,342 


19,743 
20,327 


168 
181 


Group 


9,483 
11,603 
13,360 


76.4 
15,321 


75.7 
16,368 


66,134 


349 


13,375 
13,610 


16,987 
18,732 


177 
172 


Difference 


(Impact) P-Value 


12 0.984 
1,627 * 0.053 
843 0.367 
2.9 0.346 
852 0.422 
2.5 0.428 
845 0.454 
4,180 0.271 
-704 0.619 
732 0.615 
2,756 * 0.090 
1,595 0.367 


SOURCE: MDRC calculations from unemployment insurance administrative 
records provided by the Ohio Department of Jobs and Family Services. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; 


** = 5 percent; * = 10 percent. 


@The difference between cohort subgroups is not statistically significant for 


either measure. 


Rounding may cause slight discrepancies in sums and differences. 
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Appendix Table A.6 


Impacts on Employment and Earnings, 
by Random Assignment Sector, at Towards Employment 


Health Care Manufacturing 
Difference Difference 

Outcome WA Cc (Impact) WA C (Impact) Sig. 
Year 1 
Ever employed (%) 82.1 80.4 1.7 84.2 78.2 6.0 
Earnings ($) 8,228 8,218 9 10,597 10,948 -351 
Year 2 
Ever employed (%) 75.8 74.1 1.7 82.1 73.4 8.6 ** 
Earnings ($) 11,488 10,522 966 14,771 12,902 1,869 
Year 3 
Ever employed (%) 80.0 75.1 4.9 79.9 78.6 1.4 
Earnings ($) 13,125 11,794 1,331 15,208 15,044 163 
Year 4 
Ever employed (%) 82.4 80.6 1.8 75.5 72.6 2.9 
Earnings ($) 15,397 13,211 2,186 * 16,990 17,462 -472 
Year 5 
Ever employed (%) 79.1 78.8 0.3 77.3 72.5 4.8 
Earnings ($) 15,889 14,019 1,870 18,529 18,798  -269 
Sample size (total = 698) 173 178 176 171 


SOURCES: MDRC calculations from unemployment insurance administrative records from the 
Ohio Department of Jobs and Family Services. 


NOTES: WA = WorkAdvance (program) group; C = Control group. 

A two-tailed t-test was applied to the differences between outcomes for the program and control 
groups. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Differences across subgroups were tested for statistical significance. Statistical significance 
levels (Sig.) are indicated as follows: ttt = 1 percent; tft = 5 percent; tT = 10 percent. There were 
no statistically significant impacts across subgroups. 

Sample sizes may vary because of missing values. 

Rounding may cause slight discrepancies in sums and differences. 
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Appendix Table A.7 


Impacts on Employment and Earnings in 2017 and 2018, by Site 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 
WA C Difference WA C Difference WA C_ Difference WA C_ Difference 
Outcome group group (Impact) group group (Impact) group group (Impact) group group (Impact) 
Employed (%) 
Quarter 1, 2017 72.3 70.9 1.3 67.3 65.5 1.8 69.4 67.5 1.9 73.4 72.8 0.6 
Quarter 2, 2017 75.6 71.0 46 63.3 67.8 -4.5 67.2 67.1 0.1 73.4 = 70.5 2.9 
Quarter 3, 2017 73.3 72.8 0.4 65.4 65.7 -0.4 66.2 67.5 -1.3 74.5 71.6 2.8 
Quarter 4, 2017 75.1 75.6 -0.5 68.2 65.0 3.2 66.2 69.3 -3.2 73.3 70.2 3.1 
Quarter 1, 2018 74.8 76.3 -1.5 65.7 64.6 1.1 65.4 68.9 -3.5 70.3 69.2 1.1 
Quarter 2, 2018 77.2 764 0.8 67.0 67.9 -0.9 65.9 67.9 -2.0 70.8 69.9 1.0 
Quarter 3, 2018 79.7 774 2.2 65.0 68.2 -3.2 65.7 70.4 -4.7 70.2 = 72.5 -2.3 
Quarter 4, 2018 73.4 74.5 -1.1 69.1 67.8 1.3 65.8 66.2 -0.5 70.0 71.0 -1.0 
Worked out of state 
2017 140 12.9 1.1 15.4 13.9 1.5 16.3 15.3 1.0 5.2 6.9 -1.7 
2018 16.0 14.4 1.6 15.2 13.6 1.6 17.1 14.7 2.5 6.4 5.4 1.0 
Hired at a new job 
2017 18.3 18.2 0.1 22.2 25.5 -3.3 32.2 37.3 -5.1 34.2 34.8 -0.6 
2018 20.5 20.7 -0.2 24.2 27.2 -3.0 38.9 40.9 -2.0 41.5 39.3 2.1 


(continued) 
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Outcome 


Earnings ($) 
Quarter 1, 2017 
Quarter 2, 2017 
Quarter 3, 2017 
Quarter 4, 2017 
Quarter 1, 2018 
Quarter 2, 2018 
Quarter 3, 2018 
Quarter 4, 2018 


Sample size 


WA 
group 


7,728 
7,981 
8,000 
8,842 
8,872 
9,776 
9,799 
9,957 


349 


Appendix Table A.7 (continued) 


Per Scholas 


C. Difference 


group 


6,499 
6,827 
7,143 
7,579 
7,540 
8,141 
8,067 
8,374 


341 


(Impact) 


1,229 ** 
1,154 ** 
857 
1,264 ** 
1,333 ** 
1,635 *** 
1,731 *** 
1,583 ** 


St 


WA 
group 


5,933 
5,863 
6,185 
6,563 
6,244 
6,587 
6,485 
7,355 


242 


SOURCE: MDRC calculations from National Directory of New Hires data. 


NOTES: WA = WorkAdvance; C = Control. 


Statistical significance levels are indicated as follows: *** = 1 percent; ** = 


Rounding may cause slight discrepancies in sums and differences. 


. Nicks Alliance 


C. Difference 


group 


5,539 
5,859 
5,449 
5,739 
5,632 
5,824 
5,974 
6,392 


237 


(Impact) 


WA 
group 


4,833 
4,869 
4,932 
5,105 
5,007 
5,261 
5,373 
5,606 


353 


5 percent; * = 10 percent. 


Madison Strategies Group 


C. Difference 


group 


4,290 
4,327 
4,526 
4,816 
4,818 
5,170 
5,195 
5,278 


344 


(Impact) 


543 
542 
406 
289 
189 

92 
179 
328 


Towards Employment 


WA 
group 


4,362 
4,682 
4,636 
5,326 
4,799 
4,842 
4,849 
5,251 


349 


C. Difference 


group 


4,224 
4,369 
4,341 
4,543 
4,253 
4,585 
4,595 
4,906 


349 


(Impact) 


137 
313 
295 
783 ** 
547 
258 
254 
345 


Appendix Table A.8 


Impacts on Employment and Earnings in 2017 and 2018 
by Random Assignment Cohort, by Site 


Early cohort Late cohort 
WorkAdvance Control Difference WorkAdvance Control Difference 


Outcome group group (Impact) group group (Impact) Sig. 


Per Scholas 


Employment (%) 

Quarter 1, 2017 75.1 67.6 7.4 68.7 75.0 -6.3 t 
Quarter 2, 2017 78.2 68.2 10.1 ** 72.5 74.4 -1.9 t 
Quarter 3, 2017 73.6 69.1 4.5 72.8 77.2 -4.4 

Quarter 4, 2017 75.6 72.0 3.7 74.3 80.2 -5.9 

Quarter 1, 2018 76.2 71.4 4.8 73.1 82.0 -8.9 * tt 
Quarter 2, 2018 79.3 74.1 5.2 74.6 79.3 -4.8 

Quarter 3, 2018 80.9 78.4 2.5 78.0 76.5 1.5 

Quarter 4, 2018 72.9 75.8 -2.9 73.9 73.0 0.9 


Earnings ($) 


Quarter 1, 2017 8,224 6,387 1,837 ** 7,036 6,742 294 
Quarter 2, 2017 8,513 6,498 2,015 *** 7,236 7,337 -101 t 
Quarter 3, 2017 8,107 6,678 1,429 * 7,739 7,832 -94 
Quarter 4, 2017 9,117 7,236 1,881 ** 8,476 8,029 447 
Quarter 1, 2018 8,908 7,261 1,647 ** 8,782 7,919 863 
Quarter 2, 2018 9,971 7,977 1,994 ** 9,513 8,370 1,143 
Quarter 3, 2018 10,119 8005 2,114 ** 9,367 8,196 1,171 
Quarter 4, 2018 10,233 8662 1,572 * 9,483 8,183 1,300 
Sample size 189 185 160 156 


St. Nicks Alliance 


Employment (%) 

Quarter 1, 2017 58.1 63.6 -5.5 77.4 68.0 9.4 t 
Quarter 2, 2017 58.2 68.0 -9.9 68.1 68.6 -0.6 

Quarter 3, 2017 62.0 62.8 -0.9 68.7 69.8 -1.1 

Quarter 4, 2017 65.2 62.8 2.4 70.9 68.4 2.5 

Quarter 1, 2018 61.9 62.9 -1.0 68.9 67.7 1.2 

Quarter 2, 2018 61.4 66.4 -5.0 72.2 70.8 1.4 

Quarter 3, 2018 59.9 66.4 -6.5 69.8 71.5 -1.7 

Quarter 4, 2018 62.2 69.5 -7.3 76.6 66.0 10.6 * tt 


(continued) 
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Appendix Table A.8 (continued) 


Early cohort Late cohort 
WorkAdvance Control Difference WorkAdvance Control Difference 


Outcome group group (Impact) group group (Impact) — Sig. 


Earnings ($) 


Quarter 1, 2017 5,643 5,258 385 6,139 6,010 129 
Quarter 2, 2017 5,504 5,867 = -363 6,157 5,961 195 
Quarter 3, 2017 6,213 5,228 985 6,084 5,797 287 
Quarter 4, 2017 6,196 5,439 757 6,830 6,260 570 
Quarter 1, 2018 5,698 5,023 674 6,751 6,487 264 
Quarter 2, 2018 6,001 5,312 688 7,098 6,605 493 
Quarter 3, 2018 5,832 5,571 261 7,044 6,647 398 
Quarter 4, 2018 6,900 6,356 544 7,713 6,595 1,119 
Sample size 127 131 115 106 


Madison Strategies Group 


Employment (%) 
Quarter 1, 2017 66.3 62.4 3.8 72.7 71.7 1.0 
Quarter 2, 2017 62.7 64.4 -1.7 72.0 69.1 2.9 
Quarter 3, 2017 62.0 63.9 -1.9 71.0 70.1 0.8 
Quarter 4, 2017 62.9 67.2 -4.2 69.9 70.7 -0.9 
Quarter 1, 2018 60.7 65.9 -5.2 70.7 71.0 -0.3 
Quarter 2, 2018 63.0 62.3 0.7 69.2 72.5 -3.3 
Quarter 3, 2018 63.5 65.9 -2.4 68.1 74.2 -6.1 
Quarter 4, 2018 64.3 61.4 2.9 67.5 70.3 -2.8 


Earnings ($) 


Quarter 1, 2017 4,400 3,609 792 5,238 4,922 316 
Quarter 2, 2017 4,464 3,774 690 5,287 4,801 486 
Quarter 3, 2017 4,396 4,008 388 5,520 4,925 595 
Quarter 4, 2017 4,594 4,482 112 5,669 5,048 621 
Quarter 1, 2018 4,539 4,426 113 5,497 5,137 360 
Quarter 2, 2018 4,688 4,754 -66 5,831 5,529 302 
Quarter 3, 2018 4,772 4,639 133 5,958 5,694 264 
Quarter 4, 2018 4,870 4,796 74 6,330 5,700 630 
Sample size 173 164 180 180 


(continued) 
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Appendix Table A.8 (continued) 


Early cohort Late cohort 
WorkAdvance Control Difference WorkAdvance Control Difference 


Outcome group group (Impact) group group (Impact) Sig. 


Towards Employment 


Employment (%) 
Quarter 1, 2017 68.1 65.9 2.2 78.2 80.0 -1.8 
Quarter 2, 2017 69.1 63.2 5.9 77.6 77.7 -0.1 
Quarter 3, 2017 70.0 66.9 3.0 78.9 76.3 2.6 
Quarter 4, 2017 68.0 63.1 4.9 78.0 77.8 0.3 
Quarter 1, 2018 66.5 61.7 48 73.8 77.0 -3.1 
Quarter 2, 2018 66.3 63.0 3.3 74.8 77.1 -2.4 
Quarter 3, 2018 67.0 67.5 -0.5 73.0 77.8 -4.9 
Quarter 4, 2018 66.0 66.2 -0.2 73.8 75.9 -2.1 


Earnings ($) 


Quarter 1, 2017 3,492 3,438 55 5,176 5,026 150 
Quarter 2, 2017 4,066 3,500 566 5,251 5,266 -15 
Quarter 3, 2017 4,146 3,557 589 5,078 5,160 -82 
Quarter 4, 2017 4,756 3,687 1,069 ** 5,850 5,430 419 
Quarter 1, 2018 4,337 3,358 979 ** 5,220 5,181 39 
Quarter 2, 2018 4,357 3,569 788 * 5,283 5,640 -357 T 
Quarter 3, 2018 4,231 3,622 610 5,397 5,625 -228 
Quarter 4, 2018 4,246 4,176 69 6,200 5,641 559 
Sample size 168 177 181 172 


SOURCE: MDRC calculations from National Directory of New Hires data. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig. ) 
are indicated as follows: ttt = 1 percent; tt = 5 percent; t = 10 percent. 
Rounding may cause slight discrepancies in sums and differences. 
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Appendix Table A.9 


Impacts on Unemployment Insurance Benefits in 2017 and 2018, by Site 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 
WA C Difference WA C Difference WA C_ Difference WA C_ Difference 
Outcome group group (Impact) group group (Impact) group group (Impact) group group (Impact) 
Received UI benefits (%) 
Quarter 1, 2017 0.6 0.5 0.1 1.7 0.4 1.3 2.5 5.3 -2.8 * 1.7 2.6 -1.0 
Quarter 2, 2017 7.3 5.4 1.9 7.9 4.6 3.4 3.1 4.1 -0.9 2.0 2.0 -0.1 
Quarter 3, 2017 7.4 4.2 2.9 10.3 5.9 44* 2.6 3.7 -1.0 1.7 0.8 0.9 
Quarter 4, 2017 6.6 3.5 3.1 * 7.0 6.3 0.7 3.0 4.8 -1.9 3.0 1.8 1.2 
Quarter 1, 2018 5.7 4.7 1.0 8.0 4.9 3.1 2.6 3.4 -0.8 2.5 3.0 -0.5 
Quarter 2, 2018 4.5 2.4 2.1 6.0 4.9 1.1 2.6 1.4 1.2 2.4 1.6 0.8 
Quarter 3, 2018 4.2 3.4 0.8 6.1 3.5 2.6 1.3 1.8 -0.5 2.4 1.0 1.4 
Quarter 4, 2018 2.8 5.1 -2.3 5.5 5.8 -0.2 2.2 3.3 -1.0 2.4 1.9 0.6 
Received UI benefits in 2017 10.8 8.0 2.8 13.3 10.5 2.9 6.8 9.6 -2.7 5.3 5.0 0.3 
Received UI benefits in 2018 9.6 8.4 1.2 12.4 12.3 0.1 5.7 6.1 -0.4 5.6 4.7 1.0 
Amount of UI benefits ($) 
Quarter 1, 2017 12 48 -36 33 6 27 72 106 -34 17 47 -30 
Quarter 2, 2017 172 151 20 141 95 46 77 86 -10 42 36 6 
Quarter 3, 2017 263 97 166 ** 149 122 27 69 108 -39 37 25 11 
Quarter 4, 2017 185 92 92 179 138 41 66 100 -34 45 43 2 
Quarter 1, 2018 131 105 27 128 73 55 56 78 -22 38 43 -5 
Quarter 2, 2018 128 80 48 113 69 44 51 14 36 35 25 9 
Quarter 3, 2018 104 99 6 137 79 57 20 46 -26 36 12 24 
Quarter 4, 2018 91 125 -34 64 132 -69 38 79 -40 47 36 11 


(continued) 
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Appendix Table A.9 (continued) 


Per Scholas St. Nicks Alliance Madison Strategies Group 
WA C Difference WA C_ Difference WA C_ Difference 
Outcome group group _ (Impact) group group _ (Impact) group group (Impact) 
UI benefit amount in 2017 631 388 243 501 361 140 284 401 -117 
UI benefit amount in 2018 454 408 47 441 354 87 165 217 -52 
Sample size 349 341 242 237 353 344 


SOURCE: MDRC calculations from National Directory of New Hires data. 


NOTES: WA = WorkAdvance; C = Control; Ul = Unemployment Insurance. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Rounding may cause slight discrepancies in sums and differences. 


Towards Employment 


WA C Difference 
group group (Impact) 


141 151 -11 
155 116 39 


349 349 


Appendix Figure A.1 


Impacts on Percentage Employed in an Unemployment Insurance-Covered Job 
in the Targeted Sector by Relative Quarter, Site, and Random Assignment Sector 
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Appendix Figure A.1 (continued) 


SOURCES: MDRC calculations from unemployment insurance (UI) administrative records provided by Ohio 
Department of Job and Family Services for Towards Employment sample members and Oklahoma Employment 
Security Commission for Madison Strategies Group sample members. 


NOTES: RA = random assignment; WA group = WorkAdvance (program) group. 

Sectors are defined by the North American Industry Classification System (NAICS) and are linked to employers. 
NAICS codes are not available in the UI records provided for sample members at Per Scholas and St. Nicks 
Alliance. 

Transportation includes NAICS codes starting with 48-49, manufacturing includes NAICS codes starting with 31- 
33, and health care includes NAICS codes starting with 62. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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Appendix B 


Estimating Benefits and Costs 


This Appendix describes how each of the measures of benefits and costs that were used in the 
benefit-cost analysis presented in the main text (other than program operating costs) were con- 
structed. The source of the estimates of expenditures on program costs is discussed in the main 
text. 


Impact on Pretax Earnings 


The annual impacts of the four WorkAdvance programs on pretax earnings is the key component 
of the benefit-cost analysis. These impacts were estimated for a five-and-three-quarter-year ob- 
servation period for the full samples in Per Scholas and St. Nicks Alliance, and for a five-and- 
two-quarter-year observation period for the late cohorts in Towards Employment and Madison 
Strategies Group. State unemployment insurance administrative data were used for the first three 
years for Madison Strategies Group and for the first five years for the remaining three sites, with 
administrative data from the National Directory of New Hires (NDNH) database used for the 
remainder of the observation period. Earnings impacts for the fourth year of the observation pe- 
riod were not available for Madison Strategies Group. Thus, the earnings impacts for the last two 
quarters of the third year and the first two quarters of the fifth year were used to interpolate each 
quarter of the fourth year.' After inflating the earnings impacts for the observation period to 2018 
dollars and discounting them to the 2018 base year (that is, each estimated impact for each year 
during the observation period was converted to its social value in 2018 by using the Consumer 
Price Index and a 3.5 percent social discount rate), the resulting annual values for each program 
were then simply summed during the observation period to estimate WorkAdvance’s impact on 
the total pretax earnings of participants during the observation period. The annual estimates are 
reported in Appendix Table B.1. 


Estimated regressions were used to predict the earnings impacts for the projection period 
— that is, the time beyond the observation periods for each site. These regressions are shown in 
Appendix Table B.2. The regressions used impact estimates on earnings during each calendar 
quarter for the dependent variable and the time periods, measured as the number of quarters since 
random assignment (that is, q=1, 2, 3, ...) as the independent variable. An additional independent 
variable (D) was included, which equaled one for earnings impacts estimated with the NDNH 
data and zero for impacts estimated with the UI data. Twenty-four calendar quarters of earnings 
impact estimates were available for Per Scholas and St. Nicks Alliance and 23 were available for 
Towards Employment and Madison Strategies Group. Regressions with five alternative specifi- 
cations were run (q, q + q’, q + q’ +q°, In(q), and In(q) +In(q)*). The regression with the best fit 
for each site, based on their adjusted R-Squares, F-values, and the statistical significance of their 
coefficients, was selected for purposes of predicting earnings impacts beyond 


‘As an alternative, a regression based on each quarter of available earnings impacts for Madison Strategies 
Group was estimated and used to predict the quarterly earnings impacts during the fourth year. The predicted 
earnings impacts were similar, but a little larger, than those resulting from interpolation. 
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Appendix Table B.1 


Annual Earnings Impacts, Adjusted to 2018 Dollars 
and Discounted 


Years since 

random St. Nicks Towards Madison 
assignment __ Per Scholas Alliance Employment Strategies Group 
1 $202 -$339 -$110 $1,273 
2 4,802 245 3,656 4,967 
3 6,549 -575 2,813 4,234 
4 5,762 1,431 3,100 4,012 
5 3,889 2,117 1,684 2,686 
6 4,735* 1,877* 293# 754# 


NOTES: *Based on only three calendar quarters. 
*Based on only two calendar quarters. 
Appendix Table B.2 


Regression Estimates of Changes Over Time in the Estimates of the 
Earnings Impact 


St. Nicks Towards Madison 

Per Scholas Alliance Employment _ Strategies Group 
Constant -629.3 -261.3 -452.2 -229.1 
q 277.2 33.9 187.8 203.8 
q? -10.0 = -7.4 -8.3 
D 1051.8 56.1 97.2 133.3 
Adjusted R Square 0.72 0.54 0.58 0.72 
F-value 20.3 14.5 11.0 19.5 
Number of Observations 24 24 23 23 


NOTE: All the coefficients are statistically significant to at least the 5 percent level, except 
the coefficients on D for St. Nicks Alliance, Towards Employment, and Madison Strategies 
Group. 


the observation period. The quadratic specification was the best fit for Per Scholas, Madison 
Strategies Group, and Towards Employment, implying that impacts on earnings first increased 
over time in these sites and then fell; the linear specification was the best fit for St. Nicks Alliance, 
implying that earnings impacts at this site rose over time, though slightly. The regressions predict 
that program impacts on earnings ceased to be positive at the end of the observation periods at 
Towards Employment and Madison Strategies Group, but continued to be positive for five cal- 
endar quarters beyond the observation period (that is, for seven full years in all) at Per Scholas, 
and increased by $34 each year after the observation period at St. Nicks Alliance.* When pre- 
dicted earnings impacts became negative, they were set to zero, as it is implausible that the 


>The findings for the first three programs are consistent with two earlier studies that found that changes over 
time in the impact on earnings of training and welfare-to-work programs resemble an inverted U, first increasing 
and then declining, reaching zero after four to six years: Greenberg, Michalopoulos, and Robins (2004) and 
Ashworth, Cebulla, Greenberg, and Walker (2004). An exception was found for adult women participants in 
voluntary training programs for whom impacts on earnings first appear to increase, but then not decline. About 
85 percent of the participants in Per Scholas, Madison Strategies Group, and St. Nicks Alliance were male, while 
only slightly over 40 percent of the participants in Towards Employment were male. 
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programs would have caused earnings to decrease five or six years after random assignment. 
Thus, there is no projection period for Towards Employment and Madison Strategies Group, and 
the projection period for Per Scholas is only five calendar quarters in length. Because of uncer- 
tainty concerning projections that extend for a much longer time horizon, the projection period 
for St. Nicks Alliance is limited to four years and one calendar quarter, although it is possible that 
the program at that site had positive impacts on earning for longer than 10 years. Thus, the benefit- 
cost study for St. Nicks Alliance is based on a five-and-three-quarter-year observation period and 
a four-and-one-quarter-year projection period, a total of 10 years. The predicted impacts on earn- 
ings during the projection periods for Per Scholas and St. Nicks Alliance were discounted and 
then summed along with the values obtained for the observation period. 


Fringe Benefits from Work 


According to a recent report by the U.S. Bureau of Labor Statistics,? in June 2017, wages and 
salaries averaged $24.10 per hour worked and fringe benefits equaled $11.03. Thus, fringe bene- 
fits were 46 percent of earnings ($11.03/24.10). However, this estimate, which pertains to the 
average worker in the economy, is probably high for WorkAdvance participants, as many are 
employed in jobs that pay less than average wages and probably provide fewer than average fringe 
benefits as well. Thus, the 0.46 estimate was reduced by multiplying it by 0.6 (which is roughly 
the ratio of the percentage of respondents to the WorkAdvance Year 2 survey who received paid 
sick days, a retirement plan, or eligibility for health coverage to the percentage of workers in the 
general economy who received the same fringe benefits).* The resulting figure of 0.276 was mul- 
tiplied by the impacts of the WorkAdvance programs on earnings to estimate their effects on 
fringe benefits. 


Payroll Taxes 


Payroll taxes include Social Security and Medicare taxes and taxes to support the unemployment 
insurance and worker’s compensation systems. At the federal level, the Social Security retirement 
and Medicare programs are financed by both employers and employees, while at the state level, 
unemployment insurance and worker’s compensation are financed by payroll taxes that are paid 
solely by employers. However, studies have found that once federal and state payroll taxes have 
been in place for a number of years, as is the case in the United States, most of the employers’ 
share is passed on to employees by wages that are lower than they otherwise would have been 


3U.S. Bureau of Labor Statistics (2019b). 

‘The percentage of workers in the general economy who received fringe benefits was obtained from the 
2012 Employee Benefits Survey conducted by the U.S. Bureau of Labor Statistics (Solis and Galvin 2012). Three 
separate fringe benefits were used in determining the 0.6 ratio: sick days with full pay, having a retirement plan, 
and eligibility for a health plan or medical insurance. Depending on the type of fringe benefit, the WorkAdvance 
site, and treatment or control status, the ratio varied from 0.5 and 0.7. The midpoint of 0.6 was used in the benefit- 
cost analysis. 
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paid.° This implies, in turn, that both WorkAdvance’s impact on pretax earnings and its impact 
on payroll taxes should include its impact on the employer’s share of payroll taxes, as well as its 
impact on the employee’s share of payroll taxes. 


As indicated above, the only payroll taxes that are directly paid by employees go to the 
government via the trust fund operated by the Social Security Administration (SSA). To deter- 
mine WorkAdvance’s impact on the SSA trust fund, WorkAdvance’s impacts on pretax earnings 
are multiplied by .0765, the fraction of pretax earnings that workers pay for Social Security taxes.° 


According to U.S. Bureau of Labor Statistics data, 7.6 percent of total employee com- 
pensation (that is, pretax earnings + fringe benefits + employer payroll taxes) in 2016 consisted 
of payroll taxes paid by employers to support the Social Security, Medicaid, unemployment in- 
surance, and workers’ compensation systems.’ Using these statistics, the impact of WorkAdvance 
on employer payroll taxes can be computed with the following formula: 


T = .076(E + F +T) > T =.076(E + F)/(1 - .076) 


where T is WorkAdvance’s impact on employer payroll taxes, E is the program’s impact on 
pretax earnings exclusive of payroll taxes, and F is the program’s impact on fringe benefits. Esti- 
mation of E and F is discussed above. Given these values, T was estimated and then added both 
to E to determine WorkAdvance’s impact on fotal pretax earnings and to the payroll taxes directly 
paid by WorkAdvance participants to determine the total amount of employer payroll tax. Note 
that because T appears twice in the participant column in Table 3.1 and Tables 3.3-3.5 in the main 
text, once as a positive number when it is added to E and once as a negative number when it is 
added to the participant share of payroll taxes, it cancels out in that column. 


Income Taxes 


WorkAdvance’s impacts on income tax payments by program participants are computed as the 
product of the programs’ impacts on pretax earnings and federal and state income tax rates. Fed- 
eral and state income tax rates for Per Scholas and St. Nicks Alliance participants in New York, 
Towards Employment participants in Ohio, and Madison Strategies Group participants in Okla- 
homa, were computed using version 9 of the National Bureau of Economic Research’s (2012) 
Internet TAX-SIM Model. This model produces marginal tax rates, which is appropriate because 
the income produced by WorkAdvance’s impact on wages is best viewed as marginal income. 
Because the taxable incomes of program participants vary, their marginal tax rates also vary. 
Therefore, marginal rates were computed for taxable income levels of $10,000, $15,000, $18,000, 
and $25,000. Interestingly, the rates were similar at $10,000 and $25,000, probably as a result of 
the Earned Income Tax Credit, and also higher at both of these two income levels than at $15,000 


> See, for example, the review of tax incidence studies in Hamermesh (1993). 


®It is assumed here that very few WorkAdvance participants were above the maximum earnings level that 
is subject to Social Security taxes. 


"U.S. Bureau of Labor Statistics (2018a). 
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or $18,000. Combining the federal and state rates, the ranges of rates are 14.04 to 23.95 percent 
for New York State, 12.35 to 17.65 percent for Ohio, and 15.0 to 20.25 percent for Oklahoma. 
The midpoints of these ranges, which are relatively narrow, were used in the benefit-cost study. 


Sales Taxes 


WorkAdvance’s impact on sales tax payments by program participants is computed as the prod- 
uct of the programs’ impacts on income available for consumption and state and local sales tax 
rates. This calculation assumes that all of the increase in the income of WorkAdvance participants 
was spent, rather than saved. Thus, it exaggerates the impact on sales taxes but probably not by 
much, as most WorkAdvance participants have relatively modest incomes and most low-income 
people tend to save relatively little of their incomes. Income available for consumption was com- 
puted by subtracting positive program impacts on payroll and income taxes and work-related 
expenditures and negative program impacts on transfer program benefits from pretax earnings. 
Decreases in payroll and income taxes and work-related expenditures and increases in transfer 
program benefits were instead added to pretax earnings. 


The total state and local sales tax rates for consumers in New York City (for Per Scholas 
and St. Nicks Alliance), Cleveland (for Towards Employment), and Tulsa (for Madison Strategies 
Group) was obtained from a 2012 report produced by the Government of the District of Columbia 
entitled “Tax Rates and Tax Burdens in the District of Columbia — A Nationwide Comparison,” 
which was supplemented by information on the internet. The rates were 8.875 percent in New 
York City, 8.0 percent in Cleveland, and 8.52 percent in Tulsa. 


Work-Related Expenditures 


Work-related expenditures include the costs of transportation, childcare, and uniforms that are 
required in order to work. According to the Survey of Income and Program Participation (SIPP), 
mean weekly work-related expenses other than childcare were $64.1 in 2011* and mean monthly 
earnings were $3,329.° Both figures were first annualized, and then annualized work-related ex- 
penses were divided by annualized earnings, yielding an estimate of work-related expenses per 
dollar of earnings. This computation implies that 8.34 percent of earnings are devoted to non- 
childcare work-related expenditures. Childcare expenses are not included in computing work- 
related expenses because relatively few WorkAdvance participants are working mothers with 
young children and, hence, pay for childcare. '° While some participating fathers may have paid 


SEdwards (2016). 
°U.S. Census Bureau (2019). 


”° According to the SIPP, only 31.9 percent of working mothers with children under 15 made weekly child- 
care payments in 2011. For such women, the average weekly payment was $143. See Laughlin (2013). Thus, 
the average weekly payment made by ai// working mothers with children under 15 was $46 (.319 x $143). An- 
nualizing this figure by multiplying it by 52 and then dividing it by the SIPP’s estimate of the annual earnings of 
women ($31,956) implies that 7.4 percent of the earnings of a typical working mother with children under 15 are 
used to pay for childcare. However, only 27 percent of WorkAdvance participants are female, and many of them 
do not have children of an age requiring childcare. 
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for childcare, they would be expected to be relatively few as well, as most do not live with 
spouses. 


Impacts on Transfer Program Payments (TANF, SNAP, SSI/SSDI, 
Housing Assistance, Unemployment Insurance) 


Like the data on earnings, data on the amount of unemployment insurance (UI) paid to members 
of the WorkAdvance sample, was obtained from the state governments of New York and Okla- 
homa. UI data were not obtained from Ohio. Thus, the annual impacts of WorkAdvance on the 
UI payment amounts were estimated for Per Scholas, St. Nicks Alliance, and Madison Strategies 
(but not Towards Employment) and directly incorporated into the benefit-cost analysis. 


The Year 2 survey asked respondents whether or not they received TANF, SNAP, SSI or 
SSDI, housing assistance, or UI during the month prior to the survey. The responses to these 
questions were used to estimate the impacts of WorkAdvance on the receipt of payments from 
each of these government transfer programs.'! To monetize these impacts, they were multiplied 
by the average monthly payment in the state (and when possible, the city) in which the sites were 
located during the year, corresponding as closely as possible to the second year after random 
assignment (2013 for Per Scholas and St. Nicks Alliance, and 2014 for Towards Employment 
and Madison Strategies Group). The payment amounts that were used appear in Appendix Table 
B.3. This was done for TANF, SNAP, SSI, and housing assistance, and for UI for the Towards 
Employment site. The average monthly payments for TANF, SNAP, SSI, and UI are at the state 
level while the housing assistance amounts are at the city level — New York state and city for 
Per Scholas and St. Nicks Alliance; Ohio and Cleveland for Towards Employment; and Okla- 
homa and Tulsa for Madison Strategies Group. 


After multiplying the monthly values in Appendix Table B.3 by the estimated impacts on 
the receipt of payments under each program, the resulting amounts for each program were 
summed across the programs (including UI) to obtain an estimate of the overall effect of Work- 
Advance on the total amount of government transfers paid during the month before the Year 2 
survey. Because there is no information about how this amount varied over time, it was simply 
assumed that it remained constant during each month covered by the benefit-cost analysis. Alt- 
hough this assumption undoubtably introduces errors into the analysis, it is difficult to know 
whether this causes an overstatement or an understatement of WorkAdvance’s effect on total 
expenditures on government transfer benefits. 


Administrative Cost of Transfer Programs 


The administrative costs of transfer programs include eligibility determination, developing and 
maintaining information systems, issuing payments, fraud control, staff training, and outreach to 


‘See Table 5.7 in Hendra et al. (2016). 
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Appendix Table B.3 


Monthly Transfer Program Payments 


Year 2013 2014 
Madison 
St. Nicks Towards Strategies 
Program ($) Per Scholas Alliance Employment Group 
TANF 397 397 354 188 
SNAP 274 274 253 258 
Housing Assistance 869 869 668 548 
SSI/SSDI 580 580 514 520 
Ul _ 1,380 _ 


SOURCES: For TANF information for Ohio and Oklahoma see Office of Family Assistance 
(2016), and for New York see Office of the Assistant Secretary for Planning and Evaluation 
(2015). For SNAP information see Food and Nutrition Service (2013) and Food and Nutrition 
Service (2014). For SSI/SSDI information see U.S. Social Security Administration (2019). The 
payment amount is for persons ages 18-64 and includes the average monthly payments for 
both SSI and SSDI received in December 2013 in New York and in December 2014 in Ohio 
and Oklahoma. For information on housing assistance see U.S. Department of Housing and 
Urban Development (2019). Expenditures consist of monthly HUD expenditures for eight 
housing programs: Public Housing, Housing Choice Vouchers, Moderate Rehabilitation Pro- 
gram, Project-Based Section 8, Rental Assistance Program, Section 236 Multifamily Housing, 
Section 202 Supportive Housing for the Elderly, and 811 Supportive Housing for Persons with 
Disabilities. For Ul information see Employment and Training Administration (2019). The 
monthly UI estimate for Ohio was calculated from the reports on monthly program and finan- 
cial data. The average weekly benefit of $318.38 for July 2014 was multiplied by 52 and then 
divided by 12 to compute the monthly estimate of UI. 


NOTES: TANF = Temporary Assistance for Needy Families; SNAP = Supplemental Nutrition 
Assistance Program; SSI = Supplemental Security Income; SSDI = Social Security Disability 
Income; Ul = Unemployment Insurance. 


potential beneficiaries. An estimate of WorkAdvance’s impact on the cost of administering SNAP 
is computed as the product of the estimate of the program’s impact on SNAP payments and the 
administrative costs of the SNAP program per dollar of benefits paid. The analysis uses an esti- 
mate of the latter of 15.8 percent from Isaacs (2008), which relies primarily on budget documents 
and expenditure reports that reported administrative costs as a separate line item. The administra- 
tive costs of TANF, SSI/SSDI, housing assistance, and UI were similarly computed. These five 
estimates were summed to obtain the total impact of WorkAdvance on the cost of administering 
transfer programs. The estimates of administrative costs as a percentage of benefits paid that are 
used in the benefit-cost analysis and the source of these estimates appear in Appendix Table B.4. 


Nonmarket Time 


If WorkAdvance participants work more as a result of a program, they lose time for other activi- 
ties that are presumably of value to them. The value of nonmarket time lost to study subjects as 
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Appendix Table B.4 


Administrative Costs as a Percentage of 
Benefit Payments 


Program Percentage of Benefit Source 
Payments 
TANF 15.5 percent Isaacs (2008) 
SNAP 15.8 percent Isaacs (2008) 
SSl 7.7 percent Isaacs (2008) 
Housin : HUD (2015 
RS aics e-pipercent 
Ul 12.6 percent* DOL (2018) 


NOTES: TANF = Temporary Assistance for Needy Families; 
SNAP = Supplemental Nutrition Assistance Program; SSI = 
Supplemental Security Income; Ul = Unemployment 
Insurance; HUD = U.S. Department of Housing and Urban 
Development; DOL = U.S. Department of Labor. 

*Calculation by MDRC from the information provided in 
the source. The UI figure is a national average. State-specific 
information does not appear to be available for UI. 


their work hours increase is difficult to estimate. Nonetheless, there is at least some previous re- 
search that suggests that the per-hour value of such time is substantial — at least a quarter of the 
increase in income attributable to increased hours of work, and quite probably more.'* A more 
recent random assignment study found that the value of nonmarket time is 58 percent of pretax 
earnings for telephone interviewer and data entry positions. '? For purposes of the benefit-cost 
analysis, WorkAdvance’s impact on those pretax earnings that result from reductions in non- 
market time was multiplied by 0.5 to estimate the value of this reduction. This allows for the 
possibility that because of their training, WorkAdvance participants are employed in jobs that 
are more interesting than telephone interviewer and data entry positions, so relinquishing non- 
market time is less onerous. 


While some of WorkAdvance’s impact on earnings result from increased hours of work, 
some of it results from the program’s impact on hourly wages — a human capital effect. In esti- 
mating the loss of nonmarket time, only the former needs to be considered. The top panel in 
Appendix Table B.5 addresses this by comparing WorkAdvance’s impact on earnings as a per- 
centage of control group earnings with the program’s percentage impact on the number of quar- 
ters worked. Both estimates are computed over that part of the observation period covered by the 
UI data (three years for Madison Strategies Group and five years for the remaining sites). The 
third row in the table indicates that the impact on employment accounted for only about a third 
of the increase in earnings for Per Scholas and Madison Strategies Group but played a more im- 
portant role at the other two sites. However, although the impact on earnings at St. Nicks Alliance 
was negligible early in the observation period, it was about 10 percent during the fifth year of the 


‘Bell and Orr (1994); Greenberg (1997); and Greenberg and Robins (2008). 
'3Mas and Pallais (2019). 
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Appendix Table B.5 


WorkAdvance’s Percentage Impacts on Earnings, Employment, 
Hours, and Wage Rates 


Madison 
St. Nicks Towards Strategies 
Per Scholas = Alliance# Employment Group 
Based on UI Data 
Percentage impact on earnings 20.5% *** 3.3% 12.7% 19.7% * 
Percentage impact on number of quar- 
ters employed 7.3% * 2.2% 7.3% 6.6% 
Percentage of impact on earnings at- 
tributable to impact on employment? 35.6% 66.4% 574% 33.5% 
Based on Year 2 survey and on only 
those employed 
Percentage impact on hourly 
wage rate 12.0% ** -1.5% 6.7% 8.1% 
Percentage impact on average hours per 
week -1.3% 6.9% * 6.0% 6.1% * 


NOTES: UI = Unemployment Insurance. 

aYear 5 only for St. Nicks Alliance: 9.9% for percentage impact on earnings, 2.9% for percentage impact 
on number of quarters employed, 29.9% for percentage of impact on earnings attributes to impact on em- 
ployment. 

>Percentage of impact on earnings attributable to impact on employment is calculated by dividing per- 
centage impact on earnings by percentage impact on number of quarters employed. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


observation period and the impact on employment accounts for only 30 percent of this increase. 
These fifth-year findings are important because the positive net benefit estimates for St. Nicks 
Alliance are mainly attributable to WorkAdvance’s impact on earnings late in the observation 
period. 


The bottom panel in Appendix Table B.5 uses data from the Year 2 survey, and hence is 
not directly comparable to the top panel, which is based on UI data available over three years for 
Madison Strategies Group and over five years for the remaining sites. The results for St. Nicks 
Alliance occurred before the positive impact on earnings occurred at that site and thus are proba- 
bly best ignored. The results for the other three sites suggest that there was an increase in the 
hourly wage rate, especially at Per Scholas, which is consistent with some of WorkAdvance’s 
impact in earnings being attributable to the program’s effects on human capital. The findings in 
the bottom panel also suggest that, except for Per Scholas, some of the impact on earnings is 
accounted for by increased hours of work among those who were employed. 


Based on MDRC’s interpretation of the findings in Appendix Table B.5, it is assumed in 
the benefit-cost analysis that 35 percent of the impact on earnings at Per Scholas, 50 percent of 
the earnings impact at St. Nicks Alliance and Madison Strategies Group, and 60 percent of the 
increase in earnings at Towards Employment was due to increased hours of work, with the re- 
mainder attributable to WorkAdvance’s impact on hourly wages. 
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Deadweight Loss 


The estimate of deadweight loss was obtained by multiplying an estimate of the marginal excess 
tax burden (METB) obtained from the economics literature by the estimated change in the gov- 
emmment’s fiscal position, as indicated by the total net benefit estimates in the columns for the 
government in Table 3.1 and Tables 3.3-3.5 in the main text. The METB is the increase in 
deadweight loss resulting from raising an additional dollar of tax revenue. Estimates of the METB 
are usually derived from general equilibrium models of the economy. METB estimates based on 
uncompensated labor supply elasticities, rather than compensated labor supply elasticities, are 
appropriate for the WorkAdvance benefit-cost analyses because those who support the benefit 
offset through their taxes are unlikely to be compensated through the program. A number of such 
estimates of the METB are reported in a recent book; the median value of these estimates for the 
United States is 19 cents per dollar.'* This is the value of the METB that was used to obtain the 
estimate of deadweight loss appearing in Table 3.1 and Tables 3.3-3.5. 


'4 See Table 3.2 in Boardman, Greenberg, Vining, and Weimer (2018). 
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Appendix C 


Nonmonetized Benefits and Costs 


Would net gain estimates reported in Tables 3.1 and 3.3-3.5, which are based on those benefits 
and costs that were estimated and monetized, change if the dollar values of the six items listed in 
Appendix Table C.1 could be determined? Although the precise values of the estimated net gains 
undoubtably would be different than reported, the positive conclusions about the success of the 
four WorkAdvance programs would almost certainly remain. Indeed, the net estimated gains 
might increase rather than shrink. Only two of the listed items (reductions in the earnings of 
spouses and partners, and labor market displacement) have the potential to seriously reduce the 
net gains from WorkAdvance, while three of the others should increase these gains. 


The WorkAdvance Year 2 survey found that there was some reduction in overall health 
care coverage for participants at Per Scholas and St. Nicks Alliance, but the estimated differences 
at the other two sites were positive. However, the size of these estimates was moderate and never 
statistically significant.'! Moreover, there is evidence that health itself improves with increases in 
employment and earnings,” such as those that occurred with WorkAdvance. Further offsetting 
any reductions in health insurance coverage is an indication from the Year 2 survey that the qual- 
ity of life of program participants improved at all four sites, although this impact was substantial 
and statistically significant at only the Per Scholas site.* Answers to other relevant survey ques- 
tions were also consistently more positive for WorkAdvance group members than for members 
of the control group (for example, whether they are satisfied with their current job or whether 
they often worry about their financial situation), although the differences were not usually statis- 
tically significant, except at Per Scholas. 


Impact estimates for the other four nonmonetized items listed in Appendix Table C.1 do 
not exist. Thus, some conjecture is necessary to assess them. The first of these — program impacts 
on the earnings of spouses and partners — almost certainly reduces the net gain of the families in 
which participants reside and thus the gains of society. As the earnings of participants increase, 
their spouses and partners may be able to work less. However, only around a third of participants 
lived with either a spouse or a partner at three of the sites; at the Madison Strategies Group site 
about 45 percent did. This suggests an upper bound on the extent to which spousal and partner 
earnings might have fallen, although they probably fell by far less because it is unlikely that most 
couples would be willing to relinquish the entire increase in income provided by one person so 
that the other could work less. Indeed, 87 percent of spouses and partners were not working prior 
to WorkAdvance and thus had no earnings to reduce.‘ Moreover, not all spouses and partners 
share their earnings. This may be especially likely among unmarried partners, who accounted for 
about a third to a half of all WorkAdvance couples. 


'See Appendix Table F.3 in Hendra et al. (2016). 
Fujiwara (2010). 

3See Table 5.7 in Hendra et al. (2016). 

4See Appendix Table B.1 in Hendra et al. (2016). 
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Appendix Table C.1 


Nonmonetized Benefits and Costs of WorkAdvance with Likely Magnitudes, by Site 


Per Scholas 


Participants 


Government budget 


Society 


St. Nicks Alliance 


Participants 


Government budget 


Society 


Madison Strategies Group 


Participants 


Government budget 


Society 


Has health coverage 


Moderate negative 
insignificant impact 


Moderate negative 
insignificant impact 


Moderate negative 
insignificant impact 


Moderate negative 
insignificant impact 


Negligible positive 
impact 


Negligible positive 
impact 


Quality of life 


Substantial significant 


positive impact 


Substantial significant 


positive impact 


Positive modest 
insignificant impact 


Positive modest 
insignificant impact 


Positive modest 
insignificant impact 


Positive modest 
insignificant impact 


Earnings of spouses 
and partners 


Negative impact, but 
probably modest 


Negative impact, but 
probably modest 


Negative impact, but 
probably modest 


Negative impact, but 
probably modest 


Negative impact, 
possibly modest 


Negative impact, 
possibly modest 


Effects on crime 


Likely positive, but of 
unknown magnitude 


Likely positive, but of 
unknown magnitude 


Likely positive, but of 
unknown magnitude 


Likely positive, but of 
unknown magnitude 


Likely positive, but of 
unknown magnitude 


Likely positive, but of 
unknown magnitude 


Value placed by 
public on increasing 
work among 
participants 


Positive, but of 
unknown magnitude 


Positive, but of 
unknown magnitude 


Positive, but of 
unknown magnitude 


Labor market 
displacement 


Negative, but 
probably moderate 


Negative, but 
probably moderate 


Negative, but 
probably small 


(continued) 
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Appendix Table C.1 (continued) 


Value placed by 


Has health coverage Quality of life Eaminds Ol SRousce Effects on crime RUbHereniner easing paper mane 
and partners work among displacement 
participants 
Towards Employment 
Participants : Medes! positive Small positive impact Negative impact: Dur 0 0 
insignificant impact probably modest 
Government budget 0 0 Pikely positive, pul of 0 0 
unknown magnitude 
Modest positive Negative impact, but Likely positive, but of Positive, but of Negative, but 


Society Small positive impact 


insignificant impact probably modest unknown magnitude unknown magnitude probably moderate 


SOURCE: The sources and derivation of the benefit and cost components are described in Appendix B. 


NOTE: The basis for judgements about the likely magnitudes of the nonmonetized benefits and costs are described in the text. 


As implied by a number of studies, if a training program increases employment and earnings among its participants, it might also decrease 
criminal activities among these persons.° One study found that a 1 percent increase in income reduces the propensity to commit crime by 0.6 percent 
among male U.S. youth who have permanently left school.° Although WorkAdvance participants are not exactly youths, well over half were between 
18 and 34 at study entry and, except for the sample at Towards Employment, around 85 percent were male. Thus, decreased crime among participants 
could possibly be important, resulting in benefits for the government and for the potential victims of crime, although whether this is the case is 


5See Fujiwara (2010) for a summary. 
°Grogger (1998). 


unknown. Moreover, only a limited number of persons can potentially participate in sector-fo- 
cused programs such as WorkAdvance, putting a cap on the size of these benefits. 


If the general public values reductions in the receipt of government transfer benefits by 
WorkAdvance participants, in and of itself, or the fact that these persons are more financially 
successful than they would have been without WorkAdvance, then that is a benefit to society. 
Although nothing is currently known about the possible size of this benefit, even if the value each 
member of the general public places on the success of WorkAdvance participants is very small, 
it might potentially be summed over a very large number of persons. The size of this benefit is 
capped, however, by the fact that the number of potential participants in WorkAdvance is limited. 
Moreover, according to the two-year survey, except for SNAP at the Per Scholas site, the rolls of 
government transfer programs for WorkAdvance participants fell by less than 5 percentage points 
due to the program. 


Because WorkAdvance successfully increased the earnings and employment of its par- 
ticipants, some may have ended up in jobs that otherwise would have been held by nonpartici- 
pants.’ If, as a result of such displacement, these nonparticipants became unemployed or accepted 
lower-wage jobs, then this is a potential cost of the program to society as a whole. To the extent 
that full employment was maintained in the WorkAdvance sites, however, it should have been 
relatively easy for nonparticipants to find jobs, and this cost would have been mitigated. At around 
4 or 5 percent, the unemployment rate remained low in Tulsa, the site of Madison Strategies 
Group, throughout the period covered by this benefit-cost study evaluation, suggesting that the 
displacement of nonparticipants by WorkAdvance participants was probably small. At around 6 
or 7 seven percent, the unemployment rate was initially fairly high in New York City, where Per 
Scholas and St. Nicks Alliance are located, and in the Cleveland area, the site of Towards Em- 
ployment. After a couple of years, however, the unemployment rates in these sites were similar 
to Tulsa’s. Moreover, given the specialized training provided by WorkAdvance, it is possible that 
the program imparted skills that allowed participants to leave slack labor markets for tight ones, 
making it easier for unemployed nonparticipants who remained in the slack markets to find jobs. 


TA review of studies of this effect suggests that it is moderate, perhaps under 20 percent. See Greenberg et 
al. (2011). However, studies on this topic vary greatly in methods and findings, and some suggest much larger 
effects. One recent random assignment study found that displacement offset almost the entire impact of a pro- 
gram in France that offered intensive job counseling to young, educated job seekers (Crépon et al., 2013). 
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